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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to graphical user 5 
interfaces and more particularly to graphical user inter- 
faces for pen-based computing. 
[0002] Pen-based computing is becoming increasing- 
ly popular. The user of a pen-based computer inputs 
commands and data into the computer using a pen or 10 
stylus, typically by printing or writing characters with the 
pen or stylus on the computer's display screen in a man- 
ner analogous to printing or writing with a pen on ordi- 
nary paper. Certain commands can be entered as pen 
strokes (gestures); for example, a word may be deleted *5 
by crossing it out, or a file may be opened by writing an 
M o" on its representative icon. Examples of pen-based 
computing are found in systems ranging from palm- 
sized personal digital assistants, through slate-sized 
machines such as notepad computers, and up to wall- 20 
sized interactive displays. Also, pen-based interfaces 
can sometimes be added or retrofitted to computer sys- 
tems not originally designed to be pen-based, such as 
certain personal computer systems. 

[0003] The usefulness of pen-based computing is lim- 25 
ited because of the limited capabilities of character rec- 
ognition software. In many pen-based systems, the user 
cannot write in cursive handwriting, but must print. This 
slows down input. Furthermore, the system has difficulty 
in recognizing printed characters. Consequently, the us- 
er must print carefully (viz., slowly) and often must print 
in special windows and within designated boxes, and 
the system can require special training for each individ- 
ual user. Even so, character recognition performance 
can be relatively poor. In addition, pen-based systems 
that use character recognition do not adapt well to run 
on conventional computer workstations equipped with 
a mouse, for example. 

[0004] Because of the limitations of the character-rec- 
ognition approach to pen-based input, an alternative ap- 
proach is sometimes used. In this approach, a graphical 
keyboard, visually analogous to the conventional me- 
chanical (typewriter) keyboard commonly used for input 
in non-pen-based computer systems, appears on the 
computer display. The user enters text by using the sty- 
lus to tap on desired keys of the graphical keyboard. 
[0005] The graphical-keyboard approach offers cer- 
tain advantages over the character-recognition ap- 
proach. Entering data through a graphical keyboard can 
be faster than printing (especially with numeric key- 
pads), can be less susceptible to errors when properly 
implemented (i.e., when the graphical keys are not too 
small), and does not require the system to be specially 
trained for each individual user. Moreover, the approach 
is simple to implement and integrate into existing soft- 
ware applications. 

[0006] Unfortunately, known graphical keyboards suf- 
fer from numerous drawbacks. Although the graphical 



keyboard is visually analogous to a mechanical key- 
board, it is functionally far more limited. With a graphical 
keyboard, a user cannot "touch type" using all fingers of 
both hands, but instead must "hunt and peck" with the 
pen or stylus, as if typing with one finger on a mechanical 
keyboard. Typing speed on the graphical keyboard is 
constrained by the time required for the user to move 
the finger/stylus from key to key on the keyboard. Also, 
the user cannot tap multiple keys simultaneously. In par- 
ticular, the user cannot tap a modifier key, such as the 
"shift," "Ctrl," "option," or "alt" key, at the same time as 
an ordinary key, such as a letter or number key, but in- 
stead must first tap the modifier key and then tap the 
letter or number key. In some systems the user must tap 
the modifier key once to set the modifier, tap the letter 
or number key, and then tap the modifier key again to 
release the modifier (for example, tap "shift," "a," "shift" 
to input an uppercase "A"). This can be especially awk- 
ward when multiple modifiers must be composed to form 
a single character (for example, a doubly modified char- 
acter such as ctrl-alt-x or a triply modified character such 
as ctrt-alt-X). 

[0007] Another problem with the graphical keyboard 
approach is that it places a heavy visual load on the user. 
Whereas with a mechanical keyboard, the user can 
memorize commonly used sequences of keystrokes 
and perform them "by feel" without looking at the key- 
board, with the graphical keyboard the user typically 
must look at the display before each tap of the stylus to 
ensure that the stylus is correctly placed. Furthermore, 
the graphical keyboard tends to take up a large portion 
of the screen display, in part because the keyboard con- 
tains many or all of the modifier and function keys found 
in its mechanical counterpart. 

[0008] Because of its drawbacks, the graphical-key- 
board approach to pen-based input is often considered 
cumbersome, and is used in pen-based computing sys- 
tems only as a fallback when character recognition fails 
or is insufficiently reliable. The character-recognition 
and graphical-keyboard approaches to pen-based com- 
puting are typically seen as mutually exclusive alterna- 
tives, not as complements. In particular, it has not been 
possible to combine the intuitive ease of use promised 
(if not always afforded) by the character-recognition ap- 
proach with the speed, robustness and ease of imple- 
mentation offered by the graphical keyboard approach. 
[0009] Document EP-A-0 603 669 (according to Art. 
54(3)(4) EPC relevant for novelty only) discloses a char- 
acter input method and apparatus where data can be 
input by simple operation using a keyboard displayed 
on the picture surface of a display unit. Data is input by 
designating a key position on the keyboard. Key infor- 
mation corresponds to the coordinates of a key of a 
graphic keyboard image. Path information is represent- 
ed by character patterns for a key input in a first embod- 
iment, and by starting and ending points of a path in a 
dragging-operation in a second embodiment. 
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SUMMARY OF THE INVENTION 



[0010] The present invention provides a user inter- 
face apparatus for a computing system in accordance 
to claim 1 and a method for using a graphical keyboard 5 
in accordance to claim 26. 

[0011] The present invention is a graphical keyboard 
that permits the user to do more than simple key-tap- 
ping. It lets the expert user proceed by "feel," and 
prompts the novice user when and as necessary. The 10 
graphical keyboard benefits from the expressive power 
and intuitive ease of use associated with pen strokes 
and gestures, yet does not require complex character- 
recognition software. 

[0012] The described graphical keyboard responds *5 
differently to different kinds of pen strokes. For example, 
lowercase "a" is entered by tapping a stylus on the "a" 
key of the graphical keyboard. Uppercase "A" is entered 
by an upward stroke initiated over the "a" key. Likewise, 
by stroking in other directions, a user can express other 20 
modifiers to the basic character, such as "control", "alt", 
"command," etc. Also, strokes in distinct directions can 
be used to express space, backspace, delete, and re- 
turn characters, for example. Multiple strokes can be ap- 
plied in sequence to a single key to express multiple 25 
modifiers. Visual feedback of pen strokes can be pro- 
vided in the form of marks made by the pen. Pop-up 
menus can be used to provide help with and to facilitate 
memorization of command strokes and stroke sequenc- 

[0013] In another aspect, the present invention pro- 
vides an improved method for entering characters into 
a pen-based computer using a graphical keyboard. 
[0014] A further understanding of the nature and ad- 
vantages of the invention will become apparent by ref- 35 
erence to the remaining portions of the specification and 
drawings. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0015] 



40 



Fig. 1 illustrates a sample graphical keyboard; 
Fig. 2 schematically depicts how a lowercase char- 
acter can be entered by tapping a key of the graph- 
ical keyboard of the invention; 
Fig. 3 schematically depicts how a uppercase char- 
acter can be entered by an upward stroke on a key 
of the graphical keyboard of the invention; 
Fig. 4 schematically depicts how each modifier can 50 
be distinguished by the direction of its particular 
stroke; 

Fig. 5 schematically depicts an example of how 
modifiers are combined; 

Fig. 6 schematically depicts an example of how cer- 55 
tain frequently used keys can be represented as di- 
agonal strokes; 

Fig. 7 is a series of views in which a user enters a 



lowercase character by tapping a key of the graph- 
ical keyboard; 

Fig. 8 is a series of views in which a user enters an 
uppercase character by making an upward stroke 
on a key of the graphical keyboard; 
Fig. 9 is a series of views in which a user enters a 
control character by making a rightward stroke on 
a key of the graphical keyboard; 
Fig. 10 is a series of views in which a user enters a 
control uppercase character by making a com- 
pound rightward-upward stroke on a key of the 
graphical keyboard; 

Fig. 11 depicts the two modes of selecting from a 
marking menu; 

Fig. 12 is example of a radial menu hierarchy and 
the marks that select from it; 
Fig. 13 is an example of mark confirmation; 
Fig. 14 schematically depicts a pop-up menu re- 
vealing a stroke vocabulary of eight stroke direc- 
tions; 

Fig. 15 is a series of views in which a user is pre- 
sented with and selects from a marking menu; 
Fig. 16 is a series of views showing an example of 
marking menus used to combine modifiers in the 
context of the present invention; 
Fig. 17 is a series of views showing how an eight- 
way marking menu can be used with the graphical 
keyboard of the invention; 

Fig. 18 illustrates a pen-based system suitable for 
implementing the invention; 
Fig. 19 illustrates an alternative pen-based system 
in which the display screen and input sensor are de- 
coupled; 

Fig. 19A illustrates an input sensor with an affixed 
keyboard image; 

Fig. 20 illustrates a mouse-based system suitable 
for implementing the invention; 
Fig. 21 is a flowchart of steps for responding to an 
event according to an embodiment of the invention 
that incorporates marking menus; 
Fig. 22 is a flowchart of steps for processing a Pen- 
Down event; 

Fig. 23 is a flowchart of steps for processing a Pen- 
Drag event; 

Fig. 24 is a flowchart of steps for processing a Pen- 
Still event; 

Fig. 25 is a flowchart of steps for processing a Pe- 
nUp event; 

Fig. 26 is a flowchart of steps for determining the 
modifiers corresponding to a mark; 
Fig. 27 is a flowchart of steps for determining points 
of articulation along a mark; and 
Fig. 28 is a flowchart of steps for matching points of 
articulation along a mark with centers of corre- 
sponding menus. 
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DESCRIPTION OF SPECIFIC EMBODIMENTS 

1. Introduction 

[0016] The present invention provides a system and 
method for an improved graphical keyboard that re- 
sponds differently to different kinds of pen strokes. 3y 
applying appropriate strokes, the user can easily type 
characters that on mechanical keyboards are typed by 
pressing multiple keys simultaneously. Also, the user 
can type certain frequently used keys, such as space, 
backspace, delete, and return, from anywhere on the 
keyboard. Using pen strokes makes typing easier and 
faster, reduces the visual load on the user, and makes 
it possible for the keyboard to take up less space in the 
screen display. 

2. Examples 

[0017] To understand the invention, it is helpful to be- 
gin with some illustrative examples, as depicted in Figs. 
1 -1 0. A displayed image of a sample graphical keyboard 
is shown in Fig. 1 . Tapping on any of the keys with the 
pen or stylus has the same effect as pushing the anal- 
ogous key on a mechanical keyboard. Thus to enter a 
lowercase character, a digit, or another unmodified 
character such as a period or comma, the user simply 
taps the desired key. 

[0018] To enter an uppercase character, rather than 
tapping, the user makes an upward stroke, initiated on 
the desired key. For example, as illustrated in Fig. 2, tap- 
ping on the "a" key results in "a." An upward stroke on 
"a" results in "A" (Fig. 3). 

[0019] The user accesses other modifier keys, such 
as control, alternate, option, or command, in a similar 
manner. Each modifier is distinguished by the direction 
of its particular stroke. An example mapping is shown 
in Fig. 4. According to the mapping of Fig. 4, a stroke to 
the right indicates the control ("Ctrl") modifier. Thus a 
rightward pen stroke initiated over the character "c" 
causes the graphical keyboard to output the character 
H ctr!-c." A stroke to the left indicates the option modifier, 
and a downward stroke indicates the command modifi- 
er. 

[0020] Modifiers can easily be combined. For exam- 
ple, once again applying the mapping of Fig. 4, a stroke 
to the right, starting on the character "c," followed by an 
upward stroke expresses "ctrl-C," as distinct from "ctrl- 
c." This is illustrated in Fig. 5. 

[0021] Additional stroke directions can be used to 
substitute for certain frequently used keys. For example, 
pen strokes in the diagonal directions can be used to 
express space, backspace, delete, and return. A possi- 
ble layout of these commands is shown in Fig. 6. This 
class of strokes is distinct from the modifier strokes dis- 
cussed above in connection with Figs. 3, 4, and 5, in 
that the meaning of these strokes is not affected by 
where the stroke is made. Thus a diagonal stroke up 



and to the left is interpreted in the mapping of Fig. 6 as 
a backspace regardless of whether it is made on the "a" 
key, the "c" key, or any other key of the graphical key- 
board. 

5 [0022] visual feedback can assist the user in applying 
pen strokes to the graphical keyboard. As the user 
moves the pen across the screen, a line appears on the 
screen to snow where the pen has been, as though the 
pen contained ink that left a mark on the screen. Once 

10 the pen stroke is recognized, the character is entered 
and the "ink trail" disappears. This is illustrated, for ex- 
ample, in Fig. 8. 

[0023] Each of Figs. 7-10 is a series of views showing 
how a user enters input into the graphical keyboard. The 

15 series of two views of Fig. 7 shows a user entering a 
lowercase character using the graphical keyboard. The 
user does this by tapping the desired key of the graph- 
ical keyboard with the pen or stylus. More precisely, the 
user taps the pen against the surface of the screen on 

20 which the keyboard image of the graphical keyboard is 
displayed, with the point of the pen striking the screen 
surface in proximity to the location of the desired key 
within the keyboard image. 

[0024] In the first view of Fig. 7, the user places the 
25 pen momentarily on the key corresponding to the letter 
or other character to be entered (an "a" in this example). 
The key is highlighted in the display. The character buff- 
er, which is represented by the small rectangle next to 
the view of the keyboard, pen, and user, is empty at this 
30 stage. In the second view, the user removes the pen 
from the graphical keyboard, thereby completing the 
tapping motion. The graphical keyboard produces the 
character "a" and sends it to the character buffer. The 
highlighting disappears from the display. 

35 [0025] The series of three views of Fig. 8 shows a user 
entering an uppercase character using the graphical 
keyboard of the invention. The user does this by making 
an upward stroke or mark on the desired key of the 
graphical keyboard. 

40 [0026] In the first view of Fig. 8, the user places the 
pen or stylus on the key corresponding to the character 
to be entered (an "A" in this example). The key is high- 
lighted in the display. The character buffer is empty. 
[0027] In the second view of Fig. 8, the user moves 

45 the pen "upward"-that is, toward the topmost row of the 
keyboard-while continuing to hold the pen against the 
keyboard. The user maintains contact between the pen 
point and the surface of the screen on which the key- 
board image is displayed substantially throughout the 

50 upward motion. This is called dragging the pen. The dis- 
play shows a line or mark (an ink trail) where the user 
has dragged the pen. The character buffer remains 
empty at this stage, because the character to be entered 
has not yet been recognized. The key remains highlight- 

55 ed in the display. 

[0028] In the third view of Fig. 8, the user removes the 
pen from the keyboard surface, completing the upward 
stroke. The line or mark disappears, as does the high- 
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lighting, and the character produced by the graphical 
keyboard in response to the upward stroke appears in 
the character buffer. 

[0029] The series of three views of Fig. 9 shows a user 
entering a control character using the graphical key- 
board of the invention. A "control character" is produced 
on a conventional, mechanical keyboard by holding 
down the modifier key marked "control" or "ctrl" while 
pressing a character key. Pressing the "ctrl" key while 
striking the key for the letter "a," for example, produces 
the character called "control-a," which can also be writ- 
ten as "ctrl-a" or " A a." On the graphical keyboard of the 
invention, the user can enter a control character by mak- 
ing a rightward stroke or mark on the desired key of the 
graphical keyboard. 

[0030] In the first view of Fig. 9, the user places the 
pen or stylus on the key corresponding to the character 
to be entered (the key "a" corresponding to the character 
" A a" in this example). The key is highlighted in display. 
The character buffer is empty. 

[0031] In the second view of Fig. 9, the user drags the 
pen to the right: The user moves the pen to the right 
while continuing to hold the pen against the keyboard, 
and maintains contact between the pen point and the 
surface of the screen on which the keyboard image is 
displayed substantially throughout the rightward motion. 
The display shows a line or mark where the user has 
dragged the pen. The key remains highlighted. The 
character buffer remains empty at this stage, because 
the character to be entered has not yet been recognized. 
[0032] In the third view of Fig. 9, the user removes the 
pen from the keyboard surface, completing the right- 
ward stroke. The line or mark disappears, as does the 
highlighting of the key, and the character produced by 
the graphical keyboard in response to the rightward 
stroke appears in the character buffer. 
[0033] Fig. 1 0 shows that modifiers can be combined 
by making compound strokes that combine strokes of 
different directions. The series of three views of Fig. 10 
shows a user entering a control uppercase character us- 
ing the graphical keyboard of the invention. A "control 
uppercase character" is produced on a conventional, 
mechanical keyboard by holding down the "shift" and 
"ctrl" modifier keys simultaneously while pressing a 
character key. Pressing the "shift" and "ctrl" keys while 
striking the key for the letter "a," for example, produces 
the character called "control-shift-a," which can also be 
written as "ctrl-A" or " A A." On the graphical keyboard of 
the invention, the user can enter a control uppercase 
character by making compound rightward-upward 
stroke or mark on the desired key of the graphical key- 
board. 

[0034] In the first view of Fig. 10, the user places the 
pen or stylus on the key corresponding to the character 
to be entered (the key "a" corresponding to the character 
" A A" in this example). The key is highlighted in the dis- 
play. The character buffer is empty. 
[0035] In the second view of Fig. 10, the user drags 



the pen a distance to the right and then a distance "up- 
ward"~that is, toward the topmost row of the keyboard. 
The user maintains contact between the pen point and 
the surface of the screen on which the keyboard image 
is displayed substantially throughout the rightward-and- 
then-upward motion. The display shows a line or mark 
where the user has dragged the pen. The key remains 
highlighted in the display. The character buffer remains 
empty at this stage, because the characterto be entered 
has not yet been recognized. 

[0036] In the third view of Fig. 10, the user removes 
the pen from the keyboard surface, completing the com- 
pound stroke. The line or mark disappears, as does the 
key highlighting, and the character produced by the 
graphical keyboard in response to the stroke appears in 
the character buffer. 

[0037] Although the example of Fig. 10 shows the 
rightward portion of the compound stroke preceding the 
upward portion, the same " A A" character is produced if 
the upward portion of the stroke is performed before the 
rightward portion. Other kinds of modifiers can be com- 
bined by making compound marks in other directions. 
For example, if a leftward stroke indicates the "alt" mod- 
ifier, then a leftward-upward or upward- leftward com- 
pound stroke combines the "alt" and "shift" (uppercase) 
modifiers, and a leftward-rightward or rightward-left- 
ward compound stroke combines the "ctrl" and "alt" 
modifiers. 

3. Marking Menus 

[0038] An interaction technique known as marking 
menus is used to implement the presentation of and re- 
sponse to pen strokes in the graphical keyboard of the 
present invention in the specific embodiment to be de- 
scribed. This section presents a general description of 
the technique; section 4 describes the technique more 
specifically as it applies to the present invention. A com- 
plete description of the marking menus technique can 
be found in Gordon P. Kurtenbach, "The Design and 
Evaluation of Marking Menus," Ph. D. dissertation, 
Dept. of Computer Science, Univ. of Toronto, 1993. 

3.1 Mark and Related Concepts 

[0039] The marking menus technique is intended to 
support the novice user and expert user alike, and to 
provide an efficient mechanism to support the user's 
transition from novice to expert. In marking menus, as 
a user "writes" on the computer display surface with a 
pen or stylus the computer simulates the appearance of 
ink by leaving behind a trail (commonly referred to as 
an ink-trail) of pixels whose color or shade contrasts with 
that of the background pixels. 

[0040] A mark is the series of pixels that are changed 
to the special "ink" color or shade when the pen is 
pressed and then moved across the display. The pixels 
that are changed to the ink color are those which lie di- 
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rectly under the tip of the pen as it is moved across the 
display. Freehand drawings, ranging from meaningless 
scribbles to meaningful line drawings and symbols, in- 
cluding handwriting, are examples of marks. The act of 
drawing a mark is referred to as marking. 
[0041] Marks can be created not only with a pen but 
also with other types of input devices. For example, a 
mouse can leave a trail of ink behind the tracking symbol 
when the mouse button is pressed and the mouse is 
dragged. Some systems use a pen and tablet. In this 
case, marks are made on the display by writing on the 
tablet instead of the display. Still other systems use a 
trackball. 

[0042] Marking menus find application in pen-based 
computing systems and interfaces that can interpret as 
commands and parameters the various marks a user 
makes. (This is in contrast with systems in which a us- 
er's marks cannot be recognized by the system and 
serve strictly as annotations for the user's benefit.) 
[0043] Much of the literature on pen-based computing 
refers to marks as gestures. However, the term gesture 
is inappropriate in the context of marking menus. Al- 
though it is true that creating a mark does involve a phys- 
ical gesture, the thing that is interpreted by the system 
is the mark itself. For example, the "X" mark requires a 
completely different physical gesture if performed with 
a pen instead of a mouse. Gesture is an important as- 
pect of marking because some marks may require awk- 
ward physical gestures with the input device. However, 
the two terms can be distinguished. The term mark is 
more appropriate for pen-based computer systems or 
applications that emulate paper and pen, and will be 
used hereinbelow. 

3.2 Why Use Marks? 

[0044] Conventional human-computer interfaces are 
asymmetric in terms of input and output capabilities. 
There are number of computer output modes: visual, au- 
dio and tactile. Most computers extensively utilize the 
visual mode; high-resolution images that use thousands 
of colors of can be displayed quickly and in meaningful 
ways to a user. In contrast, a computer's ability to sense 
user input is limited. Humans have a wide range of com- 
munication skills such as speech and touch, but most 
computers sense only a small subset of these. For ex- 
ample, keyboards only sense finger presses (but not 
pressure) and mice only sense very simple arm or wrist 
movements. Therefore, the advent of the pen as a com- 
puter input device provides the opportunity to increase 
input bandwidth through the use of marks. 
[0045] A major motivation for using marks is the inad- 
equacy of mouse and keyboard interfaces. This is ex- 
emplified by applications that are controlled through but- 
ton presses and position information. (The term buttons 
is used here as a generic way of describing menus 
items, dialog box items, icons, keys on a keyboard, etc., 
which are typical of direct manipulation interfaces.) Ex- 



pert users of these types of systems find the interfaces 
inadequate because button interfaces are inefficient. 
The existence of interaction techniques that override 
buttons for the sake of efficiency is evidence of this. Ex- 
5 perts, having great familiarity with the interface, are 
aware of the set of available commands. They no longer 
need menus to remind them of available commands, 
and they find that invoking commands through menu 
displays becomes very tedious. 
10 [0046] Designers have addressed this problem in 
several ways. One solution is accelerator keys which al- 
low experts direct access to commands. An accelerator 
key is a key on the keyboard which, when pressed, im- 
mediately executes a function associated with a menu 
15 item or button. The intention is that using an accelerator 
key saves the expert user the time required to display 
and select a menu item or button. Many systems display 
the names of accelerator keys next to menu items or 
buttons to help users learn and recall the associations 
20 between accelerator keys and functions. 

[0047] Accelerator keys have their problems. Arbi- 
trary mappings between accelerator keys and functions 
can be hard to learn and remember. Sometimes mne- 
monics can be established between accelerator key and 
25 function (e.g., control-o for "open"), but mnemonics 
quickly run out as the number of accelerator keys in- 
creases. Further confusion can be caused by different 
applications using a common key for different functions 
or by different applications using different accelerator 
30 keys for a common function. Experts must then remem- 
ber arbitrary or complex mappings between keys and 
functions depending on application. 
[0048] Another way of supporting the expert user is 
by supplying a command-line interface in addition to a 
35 direct manipulation interface. However, command-line 
interfaces also have problems, notably because they 
are radically different from direct manipulation interfac- 
es. To become an expert, a novice must learn another 
entirely different interface. 
40 [0049] Marks overcome the problems of accelerator 
key and command-line interface solutions. Marks, be- 
cause of their symbolic nature, can make functions more 
immediately accessible. Rather than triggering a func- 
tion by a button press, a mark can signal a command. 
45 For example, a symbolic mark can be associated with a 
function and a user can invoke the function by drawing 
the symbol. In theory, because marks can be used to 
draw any symbol or series of symbols, marks can pro- 
vide a quicker method of choosing a command than 
50 searching for a physical or graphical button and press- 
ing it. In practice, the number of marks is limited by the 
system's ability to recognize symbols and a human's 
ability to remember the set of symbols. Nevertheless, 
even if only a small set of marks are used, a user can 
55 invoke the associated functions immediately. 

[0050] Marks can also be used to hide functions. 
Functions that are not provided by means of buttons can 
nonetheless be provided by marks. This makes a full- 
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functioned interface possible with a smaller number of 
buttons, which in turn means a more efficient use of 
available display area. In many cases, moreover, a mark 
can be much more effective than a button for invoking 
a function. 

[0051] An additional motivation for using marks is that 
there are some intrinsic qualities that marks have which 
can provide a more "natural" way to express otherwise 
difficult or awkward concepts (such as spatial or tempo- 
ral information). In particular, marks provide a straight- 
forward way to embed multiple command attributes into 
a single mark. 

3.3 Marking Menus and Related Concepts 

[0052] A marking menu is an interaction technique 
that allows a user to select from a menu of items. There 
are two ways, or modes, in which a selection can be 
performed: 

(1) Menu mode. In this mode a user makes a selec- 
tion by displaying a menu. A user enters this mode 
by pressing the pen against the display and waiting 
for a short time interval, for example, approximately 
1/3 of a second. This action is sometimes referred 
to as press-and-wait. A radial menu (or pie menu) 
of items is then displayed centered around the pen 
tip. A radial menu is a menu wherein the menu items 
are positioned in a circle surrounding the cursor and 
each item is associated with a certain sector of the 
circle. A user can select a menu item by moving the 
pen tip into the sector of the desired item. The se- 
lected item is highlighted and the selection is con- 
firmed when the pen is lifted from the display. At the 
center of the radial menu is a center circie, typically, 
a circle of about 1 cm in diameter, highlighted or oth- 
erwise visually indicated. The pen can be moved 
within the center circle without causing a menu se- 
lection. 

(2) Mark mode. In this mode, a user makes a selec- 
tion by drawing a mark. A user enters this mode by 
pressing the pen against the display and immedi- 
ately moving in the direction of the desired menu 
item. Rather than displaying a menu, the system 
draws an ink-trail following the pen tip. When the 
pen is lifted, the item that corresponds to the direc- 
tion of movement is selected. 

[0053] Fig. 11 depicts two views that illustrate, respec- 
tively, the two modes of selecting a command from a 
marking menu. The first view at the left of Fig. 11 shows 
a user selecting a menu choice in menu mode. In this 
example, two menus are presented in succession to the 
user. The user chooses the "paste" menu choice by 
dragging the pen into the upper right-hand quadrant of 
the first menu, and then, without lifting the pen off the 
screen, chooses the "clipboard c" menu choice by drag- 
ging the pen into the bottom quadrant of the second 



menu. The effect of this series of choices is to cause 
data to be pasted (inserted into a working document) 
from a clipboard (storage buffer) called Clipboard C. Al- 
though the first and second menus are shown for clarity 

5 of exposition as appearing simultaneously on the dis- 
play screen in Fig. 11, in some embodiments—and in 
particular, in the embodiments of the graphical keyboard 
of the invention described herein-the first menu disap- 
pears when the second menu appears, so that the user 

10 is presented with only one menu at a time. 

[0054] The second view at the right of Fig. 11 shows 
a user selecting the same menu choice as in the first 
view, but this time doing so in mark mode. The user 
makes substantially the same pen motions and thus the 

15 substantially same mark (a diagonal upward-rightward 
line segment followed by a downward line segment) as 
in the first view, but the radial menus do not appear. The 
effect of this series of choices is once again to cause 
data to be pasted from a clipboard called Clipboard C. 

20 The second view of Fig. 11 illustrates an important con- 
cept of marking menus: The physical movement re- 
quired to select an item using a mark in mark mode mim- 
ics the physical movement involved in selecting an item 
in menu mode. 

25 [0055] Marking menus can be hierarchical. In menu 
mode, if a menu item has a subitem associated with it, 
rather than lifting the pen to select the item, the user 
waits with the pen pressed down to trigger the display 
of the submenu. The submenu is also a radial menu. 

30 The user can then select an item from a submenu in the 
manner previously described. In mark mode, a user 
makes a selection by drawing a mark where changes in 
direction correspond to selections from submenus. The 
example of Fig. 11 is an example of selecting from hier- 

35 archical menus using menu mode and mark mode. The 
first menu is at the top of the hierarchy; the second menu 
is a submenu of the first and is beneath the first in the 
hierarchy. 

[0056] Using radial menus in this way produces a set 
40 of mark which consist of a series of line segments at 
various angles ("zig-zag" marks). Marking menus which 
have no hierarchic items produce strictly straight line 
segments. Fig. 12 shows an example of a menu hierar- 
chy and its associated set of marks. The left-hand view 
45 of Fig. 12 shows the menu hierarchy. The top-level 
menu in Fig. 12 is a numeric menu containing the four 
menu choices 1, 2, 3, and 4. Each of these choices has 
associated with it an alphabetic submenu consisting of 
the choices a, b, c, and d. The right-hand view of Fig. 
50 12 shows the mark set. A mark's label indicates the 
menu items it selects. A dot indicates the starting point 
of a mark. 

[0057] It is also possible to verify the items associated 
with a mark or a portion of a mark. This is sometimes 
55 referred to as mark confirmation. In this case a user 
draws a mark but presses-and-waits at the end of draw- 
ing the mark. The system then displays radial menus 
along the mark "as if the selection were being per- 
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formed in menu mode. Fig. 13 shows an example of this 
in a menu with three levels of hierarchy. In the left-hand 
view of Fig. 1 3, the user draws the first part of the mark, 
then waits with the pen pressed down for the system to 
recognize the selection so far. In the right-hand view of 
Fig. 1 3, the system then displays its interpretation of the 
mark, and goes into menu mode for the completion of 
the selection. 

[0058] Other types of behavior can occur when select- 
ing from a marking menu, such as backing-up in a menu 
hierarchy or reselecting an item in menu mode. 

3.4 Summary 

[0059] The following is a summary of the design fea- 
tures that make marking menus a valuable and unique 
interaction technique. Marking menus: 

• Allow menu selection acceleration without a key- 
board. 

Permit acceleration on all menu items. 
Minimize the difference between the menu selec- 
tion and accelerated selection. 
Permit pointing and menu selection acceleration 
with the same input device. 
Utilize marks that are easy and fast to draw. 
Use a spatial method for learning and remembering 
the association between menu items and marks. 

• Are implementable as a "plug-in" software module. 

[0060] With conventional menus, the user moves a 
cursor (e.g., using a mouse) to a particular absolute or 
relative screen location in order to select an item from 
the menu. With marking menus, it is the direction of mo- 
tion rather than the particular location moved to that 
causes menu selection. For example, moving the cursor 
upwards on the screen can select a first menu item; 
down, a second; right, a third; left, a fourth. Additional 
selections can be associated with diagonal cursor mo- 
tions. Marking menus are arranged radially (as "pie 
menus") rather than linearly, so that each direction of 
motion is associated with a unique menu selection. 
[0061] The geometrical nature of marking menus 
makes them easy for users to memorize. A user learns 
and remembers spatial pathways through the menus. 
Such pathways can be easier to remember than combi- 
nations of accelerator keys that are conventionally used 
to substitute for menu selections. Whereas a sequence 
of accelerator keys corresponding to a sequence of se- 
lections from multiple levels of hierarchical menus can 
be somewhat arbitrary, the path traced by the pen in 
skipping over intermediate marking menus is the same 
that is used to select them when the menus are dis- 
played. In short, marking menus show the menus to the 
novice user, and leave the expert user just making the 
marks. 



4. Marking Menus in the Present Invention 

[0062] As applied in the present invention, the mark- 
ing menus technique makes menus available to assist 

5 the user who has not memorized the pen strokes, and 
hides menus from the user who does not require them. 
At any time, the user can activate a radial pop-up menu 
which shows the available choices by holding the pen 
down without moving it laterally. The user can then make 

10 the desired selection by moving the pen (that is, the 
point or tip of the pen, which is in contact with the input- 
sensing display screen or input tablet) into the "slice" of 
the pie occupied by the desired menu choice. Simply 
raising the pen off the display screen without lateral 

15 movement causes the menu to disappear and no selec- 
tion to be made. 

[0063] Figs. 14-17 depict illustrative examples of 
marking menus as applied in the context of the present 
invention. An example of a radial or pie menu corre- 
20 sponding to the eight strokes defined in the examples 
of Figs. 4 and 6 is shown in Fig. 14. The directions of 
the strokes substantially correspond to the slices of the 
pie. 

[0064] Fig. 15 is a series of views in which a user en- 

25 ters a " A a" character using the graphical keyboard of 
the invention, with marking menus used in menu mode. 
[0065] In the first view of Fig. 15, the user presses and 
holds the pen on the "a" key of the graphical keyboard. 
More precisely, the user presses and holds the pen 

30 against the display screen at the location of the "a" key 
of the keyboard image of the graphical keyboard. The 
key is highlighted in the display. Thereafter, a radial 
menu showing four choices of modifier-"uppercase," 
"option," "control," and "command"~appears on the 

35 screen, overlaid on the keyboard image and approxi- 
mately centered around the location of the tip of the pen. 
Other modifiers or other menu choices can be provided 
in alternative embodiments. Also, at this point, the user 
can simply lift the pen off the screen to cancel the menu 

40 display, in which case no character is generated by the 
graphical keyboard. 

[0066] In the second view of Fig. 15, the user moves 
the pen to the right, maintaining the pen in contact with 
the screen, to select the "control" modifier. The radial 

45 menu remains displayed. The key remains highlighted. 
[0067] In the third view of Fig. 15, the user lifts the pen 
point off the screen. This causes the "control" modifier 
to be applied to the "a" key, so that the graphical key- 
board generates the " A a" character. The radial menu dis- 

so appears, as does the key highlighting, leaving the key- 
board image as it originally appeared. 
[0068] Fig. 16 is a series of views in which a user en- 
ters a " A A" character using the graphical keyboard of 
the .invention, with marking menus used in menu mode. 

55 [0069] In the first view of Fig. 1 6, the user presses and 
holds the pen on the "a" key of the graphical keyboard. 
More precisely, the user presses and holds the pen 
against the display screen at the location of the "a" key 
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of keyboard image of the graphical keyboard. Thereaf- 
ter, a radial menu showing four choices of modifier-"up- 
percase," "option," "control," and "command "-appears 
on the screen, overlaid on the keyboard image and ap- 
proximately centered around the location of the tip of the 
pen. 

[0070] In the second view of Fig. 16, the user moves 
the pen to the right, maintaining the pen in contact with 
the screen, to select the "control" modifier. The selected 
menu choice (the "control" modifier) is highlighted. The 
radial menu remains displayed and the key remains 
highlighted. 

[0071] In the third view of Fig. 16, the user holds the 
pen still, maintaining the pen at the location previously 
reached upon selection of the "control" modifier. There- 
after, the first menu of four choices disappears and is 
replaced by a second radial menu which shows the re- 
maining available choices of modifier available to be ap- 
plied to this character~"uppercase," "option," and "com- 
mand." This second menu appears on the screen over- 
laid on the keyboard image and approximately centered 
around the location of the tip of the pen. The key remains 
highlighted , although it is partially obscured by the menu 
in this example. 

[0072] In the fourth view of Fig. 16, the user moves 
the pen upwards, that is, toward the top of the screen 
display, to select the "uppercase" modifier. The selected 
menu choice is highlighted. The radial menu remains 
displayed, and the key remains highlighted. 
[0073] In the fifth view of Fig. 1 6, the user lifts the pen 
off the screen. This causes the "control" and "upper- 
case" modifiers to be applied to the "a" key, so that the 
graphical keyboard generates the " A A" character. The 
radial menu disappears, as does the key highlighting, 
leaving the keyboard image as it originally appeared. 
[0074] Menus can be nested to levels deeper than the 
two levels shown in Fig. 16. For example, three levels 
of modifier can be applied to a character, to generate 
characters such as "comma nd-option-A" or "control-alt- 
X" using the graphical keyboard. 
[0075] Fig. 17 is a series of views showing an example 
of selection from an 8-item marking menu in an embod- 
iment of the present invention. The eight menu selec- 
tions include "uppercase" (shift), "option," "Ctrl", "com- 
mand," "backspace," "space," "return," and "delete." 
[0076] In the first view of Fig. 1 7, the character buffer 
contains a character, "A," that the user wants to delete. 
This unwanted character is immediately behind the cur- 
sor position, which is represented by the small triangle. 
The user presses and holds the pen against the display 
screen at the location of a key (here, "d") of the keyboard 
image of the graphical keyboard. The key is highlighted 
in the display. Thereafter, a radial menu showing eight 
selections, including four choices of modifie ^-"upper- 
case," "option," "control," and "command"-and four 
choices of special characters-"backspace," "space," 
"return," and (not visible in Fig. 1 7) "delete"-appears on 
the screen, overlaid on the keyboard image and approx- 



imately centered around the location of the tip of the pen. 
At this point, the user can simply lift the pen off the 
screen to cancel the menu display, in which case no 
character is generated by the graphical keyboard. 
5 [0077] In the second view of Fig. 17, the user moves 
the pen diagonally upward and to the left, in the direction 
of the "backspace" item of the radial menu, while main- 
taining the pen in contact with the screen. This selects 
the "backspace" character. The radial menu remains 
10 displayed. The key remains highlighted. 

[0078] In the third view of Fig. 1 7, the user lifts the pen 
point off the screen. This causes the graphical keyboard 
to generate the "backspace" character, which in this em- 
bodiment causes the unwanted character in the buffer 
15 to be erased and the cursor position to be moved to the 
left. (In other embodiments, the "backspace" character, 
or an equivalent such as " A h," is placed in the buffer, and 
it is left to an applications program receiving input from 
the graphical keyboard to interpret this as an erasure.) 
20 The highlighted key of the keyboard is ignored. The ra- 
dial menu disappears, as does the key highlighting, 
leaving the keyboard image as it originally appeared. 
[0079] Additional examples of marking menus as 
used in the present invention were seen in Figs 7-10. 
25 However, unlike the marking menus illustrated in Figs. 
15-17, the marking menus of Figs. 7-10 are in mark 
mode, so that no radial menus appear. 
[0080] In some embodiments of the invention, mark- 
ing menus are optional. The menus can be optionally 
30 disabled by the expert user, so that marks can be used 
without menus. In some embodiments, the marks can 
even be invisible, as though drawn in invisible ink. The 
path of the pen point (or mouse cursor or other input 
device representation) as drawn by the user is interpret- 
35 ed by the graphical keyboard as though a mark had 
been drawn, but no ink trail appears on the screen. 

5. System Components 

40 [0081] Fig. 18 illustrates a system 1 suitable for im- 
plementation of the graphical keyboard of the present 
invention. System 1 includes a processor 5, memory 6 
coupled to processor 5, user interface software 7 exe- 
cuted by processors, a pen 10, an input-sensing display 

45 device 20 that includes an input sensor 21 closely cou- 
pled to a display screen 22, a character buffer 30, and 
an event driver 40 that incorporates a delay timer 45. 
Screen 22 displays a keyboard image 25 including a plu- 
rality of keys 26. Sensor 21 , screen 22, buffer 30, and 

so event driver 40 are operationally connected to proces- 
sor 5 and are accessible to user interface software 7. 
System 1 typically further includes operating software 
(not shown) executed by processor 5 and hardware (not 
shown) for interconnecting various components of sys- 

55 tern 1 . Application program 50, which is not part of sys- 
tem 1 , accepts characters supplied by system 1 (that is, 
by the graphical keyboard) via buffer 30. 
[0082] In use, system 1 receives input in the form of 
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taps and strokes from pen 1 0 applied to input sensor 2 1 . 
Processor 5, carrying out instructions according to user 
interface software 7, determines the significance of the 
pen strokes, and in particular the characters to which 
the pen taps and marks correspond. Processor 5 out- 
puts a stream of the characters thus determined to buff- 
er 30, and optionally outputs visual feedback, such as 
ink trails and pop-up menus, to screen 22. The charac- 
ters stored in buffer 30 can then sent on for further 
processing by application program 50, which can be ex- 
ecuted by processor 5 (as shown in Fig. 18) or by an 
additional processor or processors (not shown). Appli- 
cation program 50 optionally can echo the characters 
as part of its visual display, which can also appear in 
screen 22. 

[0083] Certain elements of system 1 will now be more 
fully described. Processor 5 is a computer processing 
unit (CPU). Typically it is part of a mainframe, worksta- 
tion, or personal computer. It can include multiple 
processing elements in some embodiments. 
[0084] Pen 10 is used in conjunction with sensor 21 
to provide input to the graphical keyboard. Pen 10 can 
be any instrument suitable for use with sensor 2. For 
example, if device 20 is a touch-sensitive display 
screen, pen 10 can be a simple stylus of plastic, metal, 
wood, or other material or materials. As another exam- 
ple, if device 20 is a light-sensitive screen, pen 10 can 
be a light pen. In some embodiments, an ordinary pen 
or pencil, a laser pointer, or even the user's finger can 
serve as pen 10. 

[0085] Sensor 21 is used to detect whether pen 10 is 
in contact with device 20 and, if so, to determine the 
position of pen 10 with respect to device 20. That is, 
sensor 21 is used to determine whether pen 10 is "up" 
or "down" and where pen 10 is with respect to display 
screen 22 and in particular with respect to keyboard im- 
age 25. Sensor 21 substantially overlays (or, in some 
embodiments, underlays) screen 22 so that the position 
of pen 10 relative to sensor 21 coincides with the posi- 
tion of pen 10 relative to screen 22. 
[0086] The marking menus technique described 
above presupposes that pen 10 affects the display 
shown by screen 22 in a manner analogous to the way 
an ordinary ball-point or fountain pen affects paper. That 
is, pen 10 leaves an ink trail on screen 22. Thus when 
a user makes a stroke or a combination of strokes with 
pen 10 on input sensor 21, for example, a stroke that 
proceeds leftward and then upward, a corresponding 
mark appears on the screen, for example, an L-shaped 
mark. 

[0087] Keyboard image 25 is a graphical image gen- 
erated by processor 5 in accordance with user interface 
software 7 and displayed on screen 22. Keyboard image 
25 is shown in Fig. 18 as including keys 26 in a standard 
QWERTY keyboard layout. Other kinds of keyboards 
and layouts can be used within the scope of the inven- 
tion; for example, keyboard image 25 can include func- 
tion keys, or can be the image of a nonstandard key- 



board layout such as a Dvorak keyboard. In some em- 
bodiments, keyboard image 25 can be the image of a 
numeric keypad or a musical keyboard. 
[0088] Keyboard image 25 as shown in Fig. 18 does 

5 not include modifier keys such as "shift," "ctrl," "alt," or 
"command." According to the invention, these keys are 
unnecessary; their functionality is replaced in this em- 
bodiment by appropriate pen strokes. Thus keyboard 
image 25 occupies a smaller area of screen 20 than 

10 would be the case were the additional keys present. 
[0089] Buffer 30 stores characters that are generated 
by the graphical keyboard in response to user input. 
Typically, these characters are accepted as input by an 
application program such as application program 50. 

15 Screen 22 optionally can display ("echo") the characters 
that are stored in buffer 30 or the characters that appli- 
cation program 50 receives. Buffer 30 can be imple- 
mented using memory 6. It is accessible by both user 
interface software 7 and application program 50. 

20 [0090] Event driver 40 is connected to processor 5, 
input sensor 21, and delay timer 45, and is accessible 
by user interface software 7. Typically, event driver 40 
and delay timer 45 are implemented in operating system 
software that is executed by processor 5. They can also 

25 be implemented in hardware or partially in hardware in 
some embodiments. 

5.1 Event-driven architecture 

30 [0091] Event driver 40 is used to inform the graphical 
keyboard about certain events, or changes of state, as- 
sociated with pen 10. Specifically, event driver 40 can 
generate the following events: 

[0092] PenDown - This event is generated when pen 

35 10 is pressed down, that is, when pen 10 is newly 
brought into contact with sensor 21. 
[0093] PenDrag - This event is generated when pen 
10 is kept pressed down and the Location of pen 10 
changes. A PenDrag event is generated every time the 

40 pen moves an appreciable amount, e.g. one pixel. 
[0094] PenUp - This event is generated when pen 10 
is lifted up from (removed from contact with) sensor 21 . 
[0095] PenStill - This event is generated when pen 
10 is kept pressed on sensor 21 and the location of pen 

45 10 with respect to sensor 21 does not change a sub- 
stantial amount (typically, 2 mm) for E seconds, where 
E is a delay time. E can be a fixed time value in some 
embodiments and can be adjusted by the user in other 
embodiments. Typically, E is in the range of 

so approximately .3 seconds to 1 second. Delay timer 45 
is used to determine the length of time that pen 10 has 
been kept pressed down and held still. Timer 45 signals 
event driver 40 when the elapsed time reaches E sec- 
onds, which causes event driver 40 to generate a Pen- 

55 still event. Timer 45 begins counting when pen 10 is 
pressed down or stops moving and is reset when pen 
10 is lifted or begins moving. 

[0096] The four kinds of events are used in various 
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combinations to produce different behavior of the graph- 
ical keyboard of the invention. For example, suppose 
the user wants to enter a modified character, such as 
WA a." The user initiates the process of entering the char- 
acter by pressing the pen down on a key of the graphical 
keyboard, which causes event driver 40 to generate a 
PenDown event. The user then makes a mark by drag- 
ging the pen while holding it down. In the process of do- 
ing so, the user causes event driver 40 to generate nu- 
merous PenDrag events. Optionally, the user can pause 
during the marking to prompt the system for a menu (ei- 
ther a top-level menu or a submenu) to be displayed. 
This causes event driver 40 to generate a PenStill event. 
Finally, to complete the mark, the user lifts the pen away 
from the screen or input sensor, which causes event 
driver 40 to generate a Penllp event. Thereafter, the us- 
er's mark is processed by processor 5 according to the 
basic character determined by the pen location at the 
time of the PenDown event and the modifier(s) deter- 
mined by the mark. This typically causes the desired 
character to be output to character buffer 30. 

5.2 Software environment 

[00971 As will be appreciated by persons skilled in the 
art, the graphical keyboard of the present invention is 
typically used as part of a graphical user interface in the 
context of an operating system or environment. This en- 
vironment is supported by software that is executed by 
processor 5. Typically, the environment supports win- 
dowing operations such as associating a window with 
keyboard image 25, displaying this window at a location 
on screen 22, determining whether events associated 
with pen 10 pertain to this window and therefore to the 
graphical keyboard, directing position information from 
pen 10 to this window and thence to the graphical key- 
board, and directing the output of character buffer 30 to 
an application program such as application program 50. 
[0098] Typically, the software environment operates 
in conjunction with event driver 40 to feed to the graph- 
ical keyboard all and only those events relevant to op- 
eration of the graphical keyboard. Events that have to 
do with other windows or processes are directed to 
those other windows or processes. In such embodi- 
ments the graphical keyboard need not incorporate logic 
to discriminate between events that are relevant to its 
operation and events that are not. It is assumed here (in 
particular, in section 6 below) that the events processed 
by the graphical keyboard of the invention are relevant 
events. 

[0099] Other kinds of software environments, includ- 
ing nonwindowing environments, are contemplated 
within the scope of the invention. For example, the 
graphical keyboard of the invention can be implemented 
in a pen-based computer system that presents the user 
interface through the metaphor of a continuous sheet or 
roil of paper rather than that of a collection of discrete 
windows. 



5.3 Alternatives to Input-Sensing Screens 

[0100] In the embodiment of Fig. 18, input sensor 21 
is closely coupled to screen 22 in input-sensing display 
5 device 20. This makes the relative positions of pen 10 
and the keys 26 of keyboard image 25 immediately ap- 
parent to the user. Such an arrangement, in which the 
user effectively writes on the display, can be termed a 
direct interface. The invention can also be adapted for 
10 use with an indirect interface, in which the pen or other 
writing instrument does not contact the display. Some 
examples of such embodiments will now be given. 
[0101] In the alternative embodiment shown in Fig. 
19, sensor 21 and screen 22 are decoupled, so that a 
15 conventional, non-input-sensing display can be used. A 
digitizing tablet 70 is used to detect input from pen 10, 
and the position of pen 10 on tablet 70 is represented 
by the position of an indicator such as cursor 27 in 
screen 22. The user can determine the relative position 
20 of pen 1 0 with respect to keys 26 of keyboard image 25 
by inspecting the position of cursor 27. 
[0102] In yet another alternative embodiment, as 
shown in Fig. 20, a mouse 80 (or other equivalent input 
device) takes the place of both pen 10 and sensor 21 . 
25 The mouse position is sensed by appropriate software 
that is part of user interface software 7 executed by proc- 
essor 5. The status of mouse button 81 can ce used to 
determine whether mouse 80 is active; this is analogous 
to using sensor 21 to determine whether pen 1 0 is down 
30 or up. Cursor 27 provides visual feedback about the rel- 
ative positions of mouse 80 and keys 26 of keyboard 
image 25. Persons skilled in the art will appreciate that 
a broad range of equivalent input devices sensing two 
or more dimensions, including but not limited to track- 
35 balls and joysticks, can be used in place of mouse 80. 
[0103] In some embodiments, keyboard image 25 can 
be stenciled, glued, or otherwise affixed onto a writing 
surface in addition to or in lieu of being displayed on an 
electronic display screen. The writing surface can be, 
40 for example, digitizing tablet 70 of the embodiment of 
Fig. 19 or a mouse pad used in conjunction with mouse 
80 of the embodiment of Fig. 20. Fig. 19A illustrates a 
digitizing tablet 70' to be used in place of digitizing tablet 
70 in the embodiment of Fig. 19. Digitizing tablet 70* has 
45 stenciled on it a keyboard image 25' that includes keys 
26\ The user can interact with the graphical keyboard 
by using pen 1 0 to make strokes across keys 26' of sten- 
ciled keyboard image 25'. Stenciled keyboard image 25' 
can have a counterpart graphical keyboard image 25 
so displayed on display screen 20, with the position of pen 
10 being indicated by a cursor 27 and the pen stroke 
path indicated by an ink trail (not shown in Fig. 19). Al- 
ternatively, the on-screen visual cues (that is, cursor 27 
and keyboard image 25 including keys 26) can be omit- 
55 ted in some embodiments, in which case stenciled key- 
board image 25' replaces display 20 in communicating 
character information, position information, and the like 
to the user of the graphical keyboard of the invention, in 
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still other embodiments, stenciled keyboard image 25' 
can be applied to a dedicated portion of an input-sensing 
display screen or can be applied to a suitably construct- 
ed membrane input sensor. 

6. Method Steps 

[0104] Figs. 21-28 are flowcharts of the steps involved 
in processing input from the graphical keyboard of the 
invention in an embodiment that incorporates marking 
menus. These steps are carried out by system 1 under 
the control of processor 5, in response to PenDown, 
PenDrag, PenStill, and PenUp events generated by 
event driver 40. 

6.1 Data structures 

[0105] The event handling routines that will be de- 
scribed share certain data structures in common. These 
data structures are typically stored in memory 6 and are 
accessible by processor 5. In one embodiment the data 
structures are of the following types: 

(1) MM - represents a menu. An MM includes: 

a list of item names (e.g., shift, control, com- 
mand, option) 

a list of submenus (represented as other MMs) 
a selected item, which is an index into the list 
of item names and list of submenus 

Before processing of a character starts (that is, be- 
fore a new PenDown event occurs), a hierarchy of 
MMs exists which describes the hierarchy of menus 
that can be displayed. The MM hierarchy can be up- 
dated to reflect, in menu mode, the sequence of 
menus that has been displayed so far since the 
most recent PenDown event, or in mark mode, the 
sequence of menus that is selected by the current 
mark. 

(2) Point - a tuple (x,y,t,<t>): 



x - 

y- 
t- 



a horizontal screen location 

a vertical screen location 

the time x and y were recorded 

the angle formed by the preceding 

point, this point, and the next 



(3) PointList - an ordered list of Points. A mark is 
represented by a PointList that includes the screen 
(or input sensor) locations of the pen as the mark is 
being drawn. Each location has associated with it 
the time when it occurred (e.g., measured in milli- 
seconds from the most recent PenDown event). 

(4) ModeFlag - a toggle or flag that indicates wheth- 
er the current mode for marking menus is menu 



mode or mark mode. 
6.2 Event handling routines 

5 [0106] Fig. 21 shows the steps for responding to one 
of the four kinds of events generated by event driver 40: 
PenDown, PenDrag, PenStill, and PenUp. First the type 
of event that is to be handled is determined (Step T). 
According to the event type, one of four different event 

10 response routines is called: PenDown (Step A), Pen- 
Drag (Step B), PenStill (Step C), or PenUp (Step D). 
This completes processing until the next event occurs. 

(1 ) PenDown. If the event is a PenDown event (step 

15 A), the routine whose flowchart appears in Fig. 22 
is executed. A determination is made of which key 
of the graphical keyboard is being pressed (Step 
AA). Typically, if an input-sensing display screen is 
used for input, the key being pressed is the key with- 

20 in whose borders the tip of the pen rests. If a mouse 
or a tablet separate from the screen is used for in- 
put, the key being pressed is the key whose position 
correlates most closely with the display position of 
the cursor 27 that represents the mouse or pen. 

25 Next, the key being pressed is highlighted in the dis- 
play (Step AB). Typically this is done by darkening 
the appearance of its borders or by changing the 
color of its borders. Next, the mode is set to mark 
mode (Step AC). This is done by setting the Mode- 

30 Flag to indicate mark mode. Finally, the pen location 
and the current time are saved as a Point in a Point- 
List that is used to represent the mark currently be- 
ing made (Step AD). The tuple of values of this Point 
indicates the location of pen 10 with respect to 

35 screen 22 (or the corresponding position of cursor 
27 in embodiments that represent pen position as 
cursor position). 

(2) PenDrag. If the event is a PenDrag event (Step 
B), the routine whose flowchart appears in Fig. 23 
40 is executed. The current mode is determined by in- 
specting the ModeFlag (Step BA). If the current 
mode is mark mode, the pen location and time are 
saved as a Point in the current PointList (Step BB) 
and an ink trail is drawn from the previous pen lo- 
45 cation to the current pen location (Step BC). This 
causes a mark to appear on the display screen 
along the path of the pen as the pen is dragged. As 
the user drags the pen to make a mark, typically 
numerous PenDrag events are generated. 
50 If the current mode is menu mode, a menu is 

already on the screen. A determination is made of 
which menu item is being selected based on the pen 
location (Step BD) and this selection is stored in the 
"selected item" field of the MM that represents the 
55 current menu (Step BE). 

In menu mode, the user can select an item from 
the current menu by dragging the pen from the cent- 
er circle of the displayed menu into a radial sector 
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(i.e., a "pie slice") of the menu. As the user drags 
the pen, a series of PenDrag events occurs. The 
user can continue to move the pen, even outside 
the menu's perimeter, and the selection is main- 
tained so long as the user stays within the angular 
boundaries of the sector. Moving the pen into an- 
other sector selects a different menu item and dese- 
lects the previously selected menu item. Thus the 
menu selection just described is a revocable one. 
The selection does not cause the graphical key- 
board to generate a character or a modifier imme- 
diately. The actual application of the currently se- 
lected menu item to generate or modify a character 
occurs only on a PenUp event. 
(3) PenStill. If the event is a PenStill event (Step 
C), the routine whose flowchart appears in Fig. 24 
is executed. The current selection mode is deter- 
mined by inspecting the ModeFlag (Step CA). By 
holding the pen still, the user intends to cause men- 
us to be displayed. Thus if the current mode is mark 
mode, the mode is switched to menu mode by tog- 
gling the ModeFlag(Step CB), and a routine is 
called to determine the menu selections corre- 
sponding to the mark that has been drawn so far 
(Step CC). This routine, which is a straightforward 
extension of the routine described below with refer- 
ence to Fig. 26, updates the MM hierarchy to reflect 
the menus that have been selected and updates the 
display accordingly. The current menu (which can 
be either a top-level menu or a submenu) is dis- 
played at the pen point location. Any mark that has 
been drawn is replaced in the display with one or 
more straight line segments that represent a sche- 
matic, cleaned-up version of the mark. If the mark 
includes no substantial changes of direction, a sin- 
gle straight line segment is displayed. The line seg- 
ment has endpoints at the origin of the mark and at 
the pen point. If the mark substantially changes di- 
rection at one or more intermediate points, which 
typically indicates that the user intends to apply a 
compound modifier to the basic character, a zig-zag 
composed of distinct straight line segments is dis- 
played. Each line segment of the zig-zag corre- 
sponds to a portion of the mark that includes no sub- 
stantial changes in direction. 

If the current mode is menu mode, a menu is 
already being displayed and the user presumably 
intends to select an item from that menu. Accord- 
ingly, if the pen is positioned within a sector of the 
displayed menu (Step CD), so as to indicate that 
the user has already selected an item from the dis- 
played menu, and if there is a submenu correspond- 
ing to the indicated menu item (Step CE), the cur- 
rent menu is removed from the display (Step CF) 
and the submenu corresponding to the indicated 
menu item is displayed and becomes the current 
menu (Step CG). The MM hierarchy is updated ac- 
cordingly. The example of Fig. 16 shows how this 



appears to the user: The second view of Fig. 16 
shows a main menu being displayed, and the third 
view of Fig. 16 shows the main menu eliminated 
from the display and replaced by a submenu. If the 
pen is not within a sector of the displayed menu 
(Step CD) or there is no submenu corresponding to 
the indicated menu item (Step CE), the event re- 
sponse routine simply returns, leaving the current 
menu displayed. 

(4) PenUp. If the event is a PenUp event (Step D), 
the routine whose flowchart appears in Fig. 25 is 
executed. The current selection mode is deter- 
mined by inspecting the ModeFlag (Step DA). If the 
current mode is mark mode, any mark that has been 
drawn is erased from the display (Step DB), and the 
menu selections that correspond to the mark are 
determined according to the routine to be described 
below with reference to Fig. 26 (Step DC). After this 
determination, the MM hierarchy reflects the select- 
ed menu items. Thus the modifiers or special char- 
acters that correspond to the currently displayed 
menu and any menus previously displayed since 
the most recent PenDown event are reflected in the 
MM hierarchy. If the mode is menu mode, all menus 
and lines are erased from the display (Step DD). 
The routine for determining menu selections is not 
called in this case, because the MM hierarchy is al- 
ready up to date. 

[0107] By lifting the pen off the screen, the user in- 
tends to complete a mark and to cause the graphical 
keyboard to generate a character to be placed in the 
keyboard buffer. Accordingly (regardless of the current 
mode) the MM hierarchy is traversed to determine which 
modifiers or special characters were selected (Step 
DE). Then the appropriate character is generated ac- 
cording to and placed into character buffer 30 (Step DF). 
This character depends on the user's menu selections, 
which are encoded in the MM hierarchy. It can be a mod- 
ified (or unmodified) version of the basic character 
whose key was pressed at the most recent PenDown 
event. For example, if the basic character is "a," and the 
"ctrl" and "uppercase" modifiers are selected, the output 
character is " A A." Alternatively, if the user's mark or 
menu selections indicate a special character such as 
backspace or return, that special character is the char- 
acter to be generated, and the pressed key is ignored. 
[0108] Finally, the key that was pressed is unhighlight- 
ed, that is, restored to its normal appearance in the dis- 
play (Step DG). This completes the cycle of events, and 
the graphical keyboard is ready to accept a new char- 
acter at the next PenDown event. 

6.3 Determining Modifiers 

[0109] Fig. 26 flowcharts the steps of a routine for de- 
termining what modifiers (or special characters) corre- 
spond to a given mark drawn by the user. This routine 
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is called in Step DC of Fig. 25, as part of the processing 
of a PenUp event. The routine accepts as input a mark 
(e.g., represented as a PointList) and returns as output 
a set (possibly the empty set) of modifiers. 
[0110] Initially, the mark is tested to see if it is of ap- 
preciable length, e.g., longer than a test value such as 
5 mm (Step EA). If the mark is shorter than the test val- 
ue, it is considered not to be a mark at all, as though the 
user had simply tapped the key. In this case, the empty 
set is returned (Step EB). 

[0111] If the mark is as long or longer than the test 
value, the mark is analyzed to determine its articulation 
points (Step EC). This analysis is carried out according 
to the routine that is described below with reference to 
Fig. 27. Thereafter, the articulation points are matched 
with the centers of corresponding menus and submenus 
to determine the selected modifiers (Step ED). This 
analysis is carried out according to the routine that is 
described below with reference to Fig. 28. According to 
these "effective" or Virtual" menu selections, a set of 
modifiers is generated and returned as output (Step 
EE). 

6.4 Determining Articulation Points 

[0112] Generally speaking, the articulation points of 
a mark are the points along the mark at which the user 
indicates selections from a menu or submenu. Typically, 
this includes the first (starting) and last (finishing) points 
of the mark and any points at which the mark undergoes 
a substantial directional change. For example, if the ra- 
dial menu has eight items, so that each sector of the 
menu occupies 45 degrees, a change of greater than 
about 22.5 degrees in angular heading can be consid- 
ered substantial. Substantial angular changes are de- 
termined after the mark has been filtered for noise and 
jitter. 

[0113] More specifically, the articulation points of a 
mark are determined in one embodiment according to 
the routine whose flowchart appears in Fig. 27. This rou- 
tine, which in this embodiment is called GetArticulation- 
Points, is called in Step EC of Fig. 26. The routine ac- 
cepts as input the PointList for the current mark (the in- 
put PointList) and outputs another PointList (the output 
PointList) which contains only the articulation points. 
[0114] At the start of the routine, the input PointList is 
examined to determine whether and where the user 
paused significantly while drawing the mark (Step FA). 
This can be done, for example, by seeking pairs of 
points between which the elapsed time is greater than 
a specified delay time such as 0.5 seconds. (Typically 
the value used for the delay time here is less than the 
value used for the delay time used to trigger a PenStill 
event.) 

[0115] If at least two pause points are found (Step 
FB), then if the starting point of the mark is not included 
among the pause points, it is added to the output Point- 
List (Step FC). The pause points are then without further 



analysis taken to be articulation points and are added 
to the end of the output PointList in the order of their 
appearance in the input PointList (Step FD). The finish- 
ing point of the mark is then added to the very end of 
5 the output PointList (Step FE). This completes the rou- 
tine. 

[0116] If fewer than two pause points are found, the 
input PointList is analyzed further. First, the data of the 
PointList is preprocessed to remove certain inaccura- 

10 cies and artifacts. In particular, inaccuracies that occur 
when the pen first touches the display are removed 
(Step FF), as are inaccuracies that occur when the pen 
is finally lifted from the display (Step FG). Also, small 
angle changes due to pen jitter are filtered out (Step FH). 

15 [0117] After the PointList has been "cleaned up" in 
this manner, it is searched to detect points at which the 
mark forms a significant angle-that is, points at which 
the direction of the pen's motion changes sufficiently 
that a menu selection would be made if the menus were 

20 being displayed. This can be done, for example, by lo- 
cating Points for which the value of $ exceeds a thresh- 
old angle (Step FJ). The threshold angle typically de- 
pends on the number of menu choices. For example, if 
the marking menus support 8 choices, as is the case for 

25 the menus illustrated in the example of Fig. 14, then an- 
gular changes of greater than about 22.5 degrees can 
be considered significant and the threshold angle can 
be set to 22.5 degrees. If the menus support only 4 
choices, as is the case for the menus illustrated in the 

30 examples of Figs. 15 and 16, then the threshold angle 
can be set to 45 degrees. 

[0118] If no significant angular changes are detected 
(Step FK), the mark is considered to be a straight line, 
so that the first and last points of the mark are placed in 

35 the output PointList to be returned as the articulation 
points (Step FL). Otherwise, if one or more significant 
angular changes are detected, the mark is considered 
to include an angle or angles, so that the first point, last 
point, and the point or points of largest angular change 

40 are placed in the output PointList to be returned as the 
articulation points (Step FM). This completes the rou- 
tine. 



6.5 Matching Articulation Points with Corresponding 
45 Menus 



[0119] Generally speaking, once the articulation 
points have been found, the mark is interpreted as 
though the system had been in menu mode and the user 

so had paused each articulation point, so that the appro- 
priate menu and submenu(s) had been displayed at the 
articulation points of the mark. More specifically, in one 
embodiment the routine whose flowchart appears in Fig. 
28 is used to interpret the articulation points. This rou- 

55 tine, which in this embodiment is called Articulation- 
PointsToMenultems, is called in Step ED of Fig. 26. 
[0120] ArticulationPointsToMenultems accepts as in- 
put the PointList that contains the articulation points for 
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the current mark, which was returned as the output 
PointList from the GetArticulationPoints routine called 
in Step EC of Fig. 26, and also the hierarchy of MM data 
structures that describes the hierarchy of menus and 
submenus that can be displayed. The routine outputs 
the MM hierarchy with the appropriate selected items, 
which represents the set of modifiers. 
[0121] The routine begins by selecting the topmost 
menu from the MM hierarchy to be the current menu 
(Step GA). Thereafter the routine loops until either the 
hierarchy of menus is exhausted or the list of articulation 
points is exhausted, whichever comes first. 
[0122] More particularly, the loop proceeds as follows: 
For each articulation point in the PointList other than the 
last one (Step GB), a corresponding menu selection is 
determined (Step GC). This is the menu item that would 
be selected if the current menu were centered on the 
current articulation point and the pen were positioned at 
the location of the next articulation point in the PointList. 
The menu selection is stored as the "selected item" for 
the current menu by making appropriate modifications 
to the contents of the MM data structure that represents 
the current menu (Step GD). Then a test is made to de- 
termine whether the selected menu item has a aubmenu 
associated with it. If the selected item has no submenu 
(Step GE), the menu hierarchy is exhausted and the rou- 
tine is complete. In this case, the loop terminates imme- 
diately (Step GF). Otherwise, the submenu is taken as 
the new current menu (Step GG) and the loop repeats 
until all articulation points other than the last one have 
been processed (Step GH). The routine is then com- 
plete. Upon return from this routine, the hierarchy of MM 
data structures reflects the set of selected menu items, 
and thus the set of modifiers to be generated and re- 
turned in Step EE of Fig. 26. 

6.6 Using Event Sequences to Generate Characters 

[01 23] Sequences of events triggered by user actions 
cause the graphical keyboard to generate characters. 
For example, if a user wants to enter a lowercase char- 
acter, such as "a," the user taps the pen on the desired 
key of the keyboard image. This gives rise to the se- 
quence of events (PenDown, PenUp). This event se- 
quence, in turn, causes the steps associated with these 
events to be performed by system 1 according. to the 
logic of the flowchart of Fig. 21. More particularly, upon 
the occurrence of the PenDown event, processor 5 de- 
termines which key of keys 26 is pressed, causes that 
key to be highlighted in display screen 22, sets the mode 
to marking, and saves the location of pen 1 0. Thereafter, 
upon the occurrence of the PenUp event, processor 5 
determines that the mode is marking. Because no mark 
has been made and no modifier has been applied, proc- 
essor 5 simply unhighlights the key in display screen 22 
and places the unmodified character corresponding to 
the key into character buffer 30. 
[0124] To enter a modified character, such as "A", " A a, 
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" or " A A, M without causing menus to be displayed, the 
user touches the pen to the desired key, drags the pen 
to make the desired mark, and then lifts the pen. This 
gives rise to the sequence of events (PenDown, Pen- 

5 Drag, PenDrag, PenDrag ... PenDrag, PenUp) and 
causes the appropriate associated steps to be carried 
out. More particularly, upon the occurrence of the Pen- 
Down event, processor 5 determines which key of keys 
26 is pressed, causes that key to be highlighted in dis- 

10 play screen 22, sets the mode to marking, and saves 
the location of pen 10. Thereafter, upon the occurrences 
of the PenDrag events, processor 5 determines that the 
mode is marking, saves the current location of pen 10, 
and updates the ink trail displayed on screen 22. There- 

.15 after, upon the occurrence of the PenUp event, proces- 
sor 5 determines that the mode is marking, erases the 
ink trail from the display of screen 22, and determines 
which modifier or modifiers have been chosen accord- 
ing to the articulation points in the mark. Also, processor 

20 5 unhighlights the key in display screen 22 and places 
the modified character corresponding to the key into 
character buffer 30. 

[0125] Similarly, to generate a frequently used char- 
acter that is represented by a diagonal mark, such as 

25 backspace or return, without causing menus to be dis- 
played, the user touches the pen to any key, drags the 
pen to make the desired mark, and then lifts the pen. 
Again this gives rise to the sequence of events (Pen- 
Down, PenDrag, PenDrag, PenDrag ... PenDrag, Pen- 

30 Up) and causes the appropriate associated steps to be 
performed. 

[01 26] To use menus to enter a character having a sin- 
gle modifier, such as "A" or " A a," the user touches the 
pen to the desired key and holds it there for a sufficient 
35 time for a menu to appear (e.g., for .3 seconds). The 
user makes the desired menu selection by dragging the 
pen, and then lifts the pen. This gives rise to the se- 
quence of events (PenDown, PenStill, PenDrag, Pen- 
Drag, PenDrag ... PenDrag, PenUp), and the appropri- 
40 ate associated steps to be performed. Similarly, to gen- 
erate a frequently used character that is represented by 
a diagonal mark, such as backspace or return, using dis- 
played menus, the user touches the pen to any key, 
holds it still until the menu appears, makes the desired 
45 menu selection by dragging the pen, and then lifts the 
pen. Again this gives rise to the sequence of events 
(PenDown, PenStill, PenDrag, PenDrag, PenDrag... 
PenDrag, PenUp) and causes the appropriate associat- 
ed steps to be carried out. 
50 [0127] To use menus to enter a character having a 
double modifier, such as " A A," the user touches the pen 
to the desired key, holds it there until the menu appears, 
makes the desired selection, holds the pen still again 
until the next menu appears, makes the desired selec- 
55 tion, and lifts the pen. This gives rise to the sequence 
of events (PenDown, PenStill, PenDrag, PenDrag, 
PenDrag ... PenDrag, PenStill, PenDrag, PenDrag, 
PenDrag ... PenDrag, PenUp) and causes the appropri- 
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ate associated steps to be performed. Similarly, the user 
can add yet further modifiers using menus by continuing 
to drag and then hold still the pen as needed. Thus to 
use menus to enter a triply modified character, such as 
M command-option-A," the user touches the pen to the 
desired key, holds it there until the menu appears, 
makes the desired selection, holds the pen still again 
until the next menu appears, makes the desired selec- 
tion, holds the pen still yet again until the next menu ap- 
pears, makes the desired selection, and lifts the pen. 
This gives rise to the sequence of events (PenDown, 
PenStill, PenDrag, PenDrag, PenDrag ... PenDrag, 
PenStill, PenDrag, PenDrag, PenDrag... PenDrag, 
PenStill, PenDrag, PenDrag, PenDrag ... PenDrag, Pe- 
nUp) and causes the appropriate associated steps to be 
performed. 

[0128] The user can also use the mark confirmation 
feature of marking menus to obtain a menu display dur- 
ing the process of making a mark representing a com- 
pound modifier. For example, if the user touches the pen 
to a key, drags the pen to the right and then upward (sub- 
stantially without pausing during the change of direc- 
tion), and then holds the pen still without lifting it away 
from the screen, thereafter the menu for the rightward 
mark and submenu for the upward mark are both dis- 
played. The sequence of events for this example is 
(PenDown, PenDrag, PenDrag, ... PenDrag, Pendrag, 
PenStill). Once again, the appropriate associated steps 
are carried out. 

[0129] Still other combinations of events can be sup- 
ported in some embodiments. For example, in some 
embodiments a modifier can be applied to a character 
generated by a diagonal stroke. 

7. Discussion of the Software 

[0130] A current embodiment of the invention is im- 
plemented in software on a digital computer. A source 
code for a software program for implementation of this 
embodiment which supports singly modified characters, 
includes a single source code file. It is written in the Hy- 
perCard scripting language for the Apple Macintosh 
family of computers. Persons of skill in the art will ap- 
preciate that a wide variety of programming languages 
and hardware configurations can readily be used based 
on this disclosure. 

8. Conclusion 

[0131] The present invention provides a simple and 
easy-to-implement technique that enables commands 
and data, such as alphanumeric data, to be entered ac- 
curately and efficiently into a pen-based computer via a 
graphical keyboard. The graphical keyboard allows a 
user to make use of existing knowledge of keyboard en- 
try. The use of strokes and marking menus provides ac- 
celerated input. Because of the self-revealing nature of 
marking menus, these accelerators are accessible to 



novices. They are also valuable to experienced users 
who may need to refamiliarize themselves with the men- 
us from time to time, for example, if larger menus with 
a greater variety of commands are installed as part of a 

5 software upgrade to the graphical keyboard. 

[01 32] The graphical keyboard of the invention offers 
improved ease of use over existing graphical key- 
boards. For example, the graphical keyboard of the in- 
vention is modeless. In previous graphical keyboards, 

10 tapping a modifier key set a mode (e.g., shift mode) that 
explicitly had to be released. This inconvenience and 
awkwardness is avoided in the present invention. More- 
over, the graphical keyboard of the invention can be 
made smaller in terms of screen display size than pre- 

15 vious graphical keyboards, inasmuch as modifier keys 
and certain command keys such as space and delete 
need not be explicitly represented. Still further, the 
graphical keyboard of the invention allows users to pro- 
ceed "by feel"-that is, to apply multiple modifiers and to 

20 input stroke-based commands without looking at the 
displayed keyboard every step of the way. Numerous 
other advantages of the invention will be apparent to 
those of skill in the art. 

[0133] The invention has relevance to most pen- 
25 based and stylus-based computing systems. :n particu- 
lar, the invention lends itself to easy retrofit to existing 
applications. All that is needed for such a retrofit is to 
build a module that the computer system sees as equiv- 
alent to the standard mechanical keyboard. 
30 [0134] The invention has been explained with refer- 
ence to specific embodiments. Other embodiments are 
contemplated without departing from the scope of the 
invention as claimed. For example, the basic character 
to be modified by a mark can be designated according 
35 to the finishing point rather than the starting point of the 
mark. The mark can also be decoupled from the key 
designation; for example, the user can indicate the key 
with a tap to the screen and then draw the mark sepa- 
rately. Different kinds of strokes, such as double taps 
40 (two taps of the stylus in rapid succession, analogous 
to the "double click" of a mouse button), can be used 
with the invention. Likewise, pen pressure or velocity 
can be used to encode command information. Different 
kinds of commands can be implemented, such as font 
45 size commands. Alternate menu layouts can be used 
for pop-up menus. Menus need not be organized hier- 
archically. Different kinds of graphical keyboards and 
keyboard layouts can be used. Moreover, the invention 
can be applied in systems other than those ordinarily 
50 thought of as pen-based; in particular, it finds application 
in systems in which a finger, a mouse, a laser beam, or 
any one of myriad other instruments analogous to a pen 
is used in lieu of a mechanical keyboard to input char- 
acters. Still further variations and extensions will be ap- 
55 parent to those of skill in the art within the scope of the 
invention as claimed. It is therefore not intended that this 
invention be limited, except as indicated by the append- 
ed claims. 
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Claims 



1 . A user interface apparatus for a computing system 
(1) having 

a display device (20) to display a graphical key- 
board image (25), 

an input source comprising a designating de- 
vice (10), said input source providing an input 
comprising key information corresponding to a 
key designated on said graphical keyboard im- 
age by said designating device and path infor- 
mation corresponding to at least one direction 
of displacement of said designating device on 
said graphical keyboard image, and 

an output generator, responsive to said path in- 
formation provided by said input source, to pro- 
duce keycode data defining an output character 
by applying a modifier corresponding to said 
path information to an input character corre- 
sponding to said key information, with said 
modifier being determined according to said at 
least one direction of displacement. 

2. The apparatus according to claim 1 , wherein said 
graphical keyboard image (25) comprises a repre- 
sentation of a plurality of characters, and said key 
information provided by said input source compris- 
es a character to which said key corresponds. 

3. The apparatus according to any one of the claims 
1 or 2, wherein said input source further comprises 
a sensor (22) responsive to a plurality of positions 
of said designating device (10), said sensor provid- 
ing said input source with position information about 
said designating device (10) during a manipulation 
of said designating device (10), said position infor- 
mation determining said direction of displacement. 



plurality of positions. 

7. The apparatus according to any one of claims 1 to 
6, wherein said modifier is selected from the group 

5 consisting of shift, uppercase, control, alt, option, 
and command. 

8. The apparatus according to any one of the claims 
1 to 7, wherein said output generator produces said 

10 output by determining a character corresponding to 
said path information and independently of said key 
information. 

9. The apparatus according to claim 8, said character 
15 being a special character selected from the group 

consisting of return, space, backspace, and delete. 

10. The apparatus according to any one of the claims 
1 to 9, further comprising a buffer (30) for storing 

20 said output character. 



11. The apparatus according to any one of the claims 
1 to 10, said input source providing said key infor- 
mation according to a position of said designating 

25 device (10) relative to said graphical keyboard im- 
age (25). 

12. The apparatus according to any one of the claims 
1 to 11, said input source providing said key infor- 

30 mation according to one point of a series of points 
corresponding to movement of said designating de- 
vice. 

13. The apparatus according to claim 12, said one point 
35 being a starting point of said series. 

14. The apparatus according to any one of the claims 
1 to 1 3, said display device further displaying a rep- 
resentation of said path information. 



40 



4. The apparatus according to claim 3, wherein said 
sensor (22) is integrated with said display (20), said 
position information includes position data in a vi- 
cinity of said graphical key board image throughout 
a substantial portion of said manipulation of said 
designating device (10). 

5. The apparatus according to any one of the claims 
1 to 4, wherein said designating device (10) com- 
prises a pointing device selected from the group 
consisting of a pen, a light pen, a stylus, a pencil, a 
pointer, a mouse, a trackball, a joystick, or a finger. 

6. The apparatus according to any one of the claims 
1 to 5, wherein said designating device comprises 
a cursor displayed by said display device (20), said 
cursor providing a representation of a position of a 



15. The apparatus according to claim 14, wherein said 
representation of said path information comprises 
a mark corresponding to said path information, said 
mark being displayed in conjunction with said 

45 graphical keyboard image. 

16. The apparatus according to claim 14, said repre- 
sentation of said path information comprising an ink 
trail corresponding to said series of points displayed 

so in the vicinity of said graphical keyboard image. 

17. The apparatus according to claim 14, said repre- 
sentation of said path information comprising a 
menu displayed in conjunction with said graphical 

55 keyboard image. 

18. The apparatus according to claim 1, wherein said 
input source includes a sensor, responsive to a plu- 
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rality of positions of said designating device relative 
to said sensor, with said generator including a proc- 
essor, coupled to said sensor and said display, to 

display, in conjunction with said display, said 
graphical keyboard image, 

receive from said sensor an input comprising 
position information and said path information, 
with said image including a plurality of keys, 
each such key being associated with a location 
in said image, said position information com- 
prising a selected position of said plurality of 
positions of said designating device, and said 
path information comprising a series of posi- 
tions of said plurality of positions of said posi- 
tion designator, said series representing said 
direction of displacement, 

correlate said selected position with a location 
in said image to determine a selected location, 

associate said selected location with a key of 
said plurality of keys to determine a selected 
key, 

interpret said path information to determine a 
command, and 

generate an output, said output being selected 
from the group consisting of a character deter- 
mined by applying said command to said se- 
lected key and a character determined through 
execution of said command independently of 
said selected key 

19. The apparatus according to claim 18, wherein a 
buffer is coupled to said processor to receive said 
output thus generated. 

20. The apparatus according to any one of the claims 
18 or 19, wherein an event driver is coupled to said 
sensor and said processor, to generate events re- 
sponsively to changes in said relative position of 
said designating device with respect to said sensor, 
to substantial changes in a proximity between said 
designating device and said sensor, and to substan- 
tial pauses in a relative motion of said designating 
device with respect to said sensor, and wherein said 
processor generates said output responsively to an 
event generated by said event driver. 

21. The apparatus according to any one of the claims 
1 8 to 20, wherein said processor, in conjunction with 
said display, provides a visual representation of 
said path information in conjunction with said graph- 
ical keyboard image. 



12. The apparatus according to claim 1, wherein said 
display comprises a substantially planar screen to 
display a graphical keyboard image comprising a 
plurality of keys, with said designating device in- 
cluding a pen to designate a plurality of positions 
relative to said graphical keyboard image, said input 
source including a sensor, coupled to said display, 
to sense positions designated by said pen, a buffer 
to receive an output, and said generator including 
a processor, coupled to said display screen, said 
sensor, and said buffer, to generate a character re- 
sponsively to a series of positions designated by 
said pen, said series of positions describing said di- 
rection of displacement as a relative motion of said 
pen with respect to said graphical keyboard image 
and substantially in a surface corresponding to the 
plane of said display screen. 

23. The apparatus according to claim 22, wherein said 
display screen and said sensor are physically cou- 
pled in an input-sensing display, with said surface 
being substantially planar and substantially parallel 
to said plane of said display screen. 

24. The apparatus according to any one of the claims 
22 or 23, wherein said sensor overlies said display 
screen. 

25. The apparatus according to any one of the claims 
22 to 24, wherein said sensor underlies said display 
screen. 

26. A method for using a graphical keyboard having a 
step of displaying a graphical keyboard image via a 
display and comprising the steps of: 

using an input source comprising a designating 
device to provide an input comprising key infor- 
mation and path information, said key informa- 
tion corresponding to a key designated on said 
graphical keyboard image by said designating 
device, said path information corresponding to 
at least one direction of displacement of said 
designating device on said graphical key board 
image; and 

using an output generator to produce, respon- 
sively to said key information and said path in- 
formation, keycode data defining an output 
character by applying a modifier corresponding 
to said path information to an input character 
corresponding to said key information, with said 
modifier being determined according to said at 
least one direction of displacement. 

27. The method of claim 26, wherein said step of using 
said output generator to produce an output respon- 
sively to said path information is further including 
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the steps of using said output generator to examine 
said series of points to determine a set of articula- 
tion points, and using said output generator to de- 
termine a command according to said set of articu- 
lation points. 

28. The method of claim 26, further comprising the 
steps of providing a computing system including a 
processor, a sensor, a display, and a buffer, a meth- 
od for using a graphical keyboard to produce a char- 
acter, wherein said step of using an input source 
comprises a step of sensing with said sensor a mo- 
tion corresponding to the at least one direction of 
displacement of the designating device relative to 
said sensor by maintaining said designating device 
proximate to said sensor throughout a substantial 
portion of said motion, and receiving in said buffer 
said keycode thus generated. 

29. The method of claim 28, wherein said step of using 
said processor to generate said character includes 
the steps of determining an unmodified character 
according to an initial position of said designating 
device at a starting point of said motion, determining 
a modifier according to a path taken by said desig- 
nating device during said motion, and applying said 
modifier to said unmodified character. 

30. The method of claim 28, wherein said step of using 
said processor to generate said character includes 
the step of determining a special character accord- 
ing to a path taken by said designating device dur- 
ing said motion and independently of an initial po- 
sition of said pen device at a starting point of said 
motion. 

31 . The method of claim 26 further comprising the steps 
of displaying a menu having a plurality of items, pro- 
viding said graphical keyboard image with a repre- 
sentation of a plurality of keys, using said designat- 
ing device to designate an item from said menu, and 
using said processor to generate a character ac- 
cording to said designated menu item, with said key 
information corresponding to a key from said plural- 
ity of keys designated by said designating device, 
defining a designated key. 

32. The method of claim 31, wherein said step of dis- 
playing a menu comprises displaying a radial menu 
comprising a representation of a set of modifiers, 
said step of using said designator to designate an 
item comprises a step of selecting a modifier from 
said set modifiers, and said step of using said proc- 
essor to generate said character includes a step of 
applying said selected modifier to a character cor- 
responding to said designated key. 

33. The method of claim 31, wherein said step of dis- 



playing a menu comprises displaying a radial menu 
comprising a representation of a set of special char- 
acters, said step of using said designator to desig- 
nate an item comprises a step of selecting a special 
character from said set of special characters, and 
said step of using said processor to generate said 
character comprises a step of producing said se- 
lected special character independently of said des- 
ignated key. 
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Patentanspruche 

1. Benutzerschnittstellenvorrichtung fur ein Rechner- 
system (1), mit: 

einer Anzeigevorrichtung (20) zum Anzeigen 
einer graphischen Tastaturabbildung (25), 

einer Eingabequelle, welche ein Zeigegerat 
(10) umfafit, wobei die Eingabequelle eine Ein- 
gabegrdfie bereitstellt, die Tasten information 
umfa&t, welche einer Taste entspricht, die auf 
der graphischen Tastaturabbildung durch das 
Zeigegerat und eine Pfadinformation, die min- 
destens einer Verschiebungsrichtung des Zei- 
gegerates auf der graphischen Tastaturabbil- 
dung entspricht, bezeichnet wird, und 

einem Ausgangsgenerator, welcher auf die von 
der Eingangsquelle bereitgestellte Pfadinfor- 
mation reagiert, indem er ein Ausgabezeichen 
definierende Tasten coded ate n erzeugt, indem 
ein der Pfadinformation entsprechender Modi- 
fikator auf ein der Tasten information entspre- 
chendes Eingangszeichen angewendet wird, 
und der Modifikator anhand der mindestens ei- 
nen Verschiebungsrichtung bestimmt wird. 

40 2. Vorrichtung nach Anspruch 1 , wobei die graphische 
Tastaturabbildung (25) eine Darstellung einer Viel- 
zahl von Zeichen umfaRt, und die von der Eingabe- 
quelle bereitgestellte Tasteninformation ein Zei- 
chen umfalit, welchem die Taste entspricht. 

45 

3. Vorrichtung nach einem der Anspruche 1 oder 2, 
wobei die Eingabequelle ferner einen auf eine Viel- 
zahl von Positionen des Zeigegerates (10) reagie- 
renden Sensor (22) umfaftt, wobei der Sensor die 
so Eingabequelle mit Positionsinformation uber das 
Zeigegerat (10) wahrend einer Manipulation des 
Zeigegerates (10) versorgt und wobei die Positions- 
information die Verschiebungsrichtung bestimmt. 

55 4. Vorrichtung nach Anspruch 3, wobei der Sensor 
(22) in die Anzeige (20) integriert ist, die Positions- 
information Positionsdaten in einer Nahe der gra- 
phischen Tastaturabbildung uber einen wesentli- 
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chen Abschnitt der Manipulation des Zeigegerates 
(10) enthalt. 

5. Vorrichtung nach einem der Anspruche 1 bis 4, wo- 
bei das Zeigegerat (10) eine aus der aus einem 
Schreibgerat, einem Lichtstift, einem Stift, einem 
Fuller, einem Zeigegerat, einer Maus, einem Track- 
ball, einem Joystick oder einem Finger bestehen- 
den Gruppe ausgewahltes Punktzeigegerat um- 



6. Vorrichtung nach einem der Anspruche 1 bis 5, wo- 
bei das Zeigegerat einen durch die Anzeigevorrich- 
tung (20) angezeigten Cursor umfafit, wobei der 
Cursor eine Darstellung einer Position einer Viel- 
zahl von Positionen bereitstellt. 

7. Vorrichtung nach einem der Anspruche 1 bis 6, wo- 
bei der Modifikator aus der aus Shift, Uppercase, 
Control, Alt, Option und Command bestehenden 
Gruppe ausgewahlt wird. 

8. Vorrichtung nach einem der Anspruche 1 bis 7, wo- 
bei der Ausgabegenerator die Ausgabegrdfie durch 
Ermitteln eines der Pfadinformation entsprechen- 
den Zeichens und unabhangig von der Tasteninfor- 
mation erzeugt. 

9. Vorrichtung nach Anspruch 8, wobei das Zeichen 
ein aus der aus Return, Space, Backspace und De- 
lete bestehenden Gruppe ausgewahltes spezielles 
Zeichen ist. 

10. Vorrichtung nach einem der Anspruche 1 bis 9, wel- 
che femer einen Puffer (30) fur die Speicherung des 
ausgegebenen Zeichens umfalit. 

11. Vorrichtung nach einem der Anspruche 1 bis 10, 
wobei die Eingabequelle die Tasteninformation ge- 
mali einer Position des Zeigegerates (10) in bezug 
auf die graphische Tastaturabbildung (25) bereit- 
stellt. 

12. Vorrichtung nach einem der Anspruche 1 bis 11 , wo- 
bei die Eingabequelle die Tasteninformation gemali 
einem Punkt einer Folge von der Bewegung des 
Zeigegerates entsprechenden Punkten bereitstellt. 

13. Vorrichtung nach Anspruch 12, wobei der eine 
Punkt ein Startpunkt der Folge ist. 

14. Vorrichtung nach einem der Anspruche 1 bis 13, 
wobei die Anzeigevorrichtung ferner eine Darstel- 
lung der Pfadinformation anzeigt. 

15. Vorrichtung nach Anspruch 14, wobei die Darstel- 
lung der Pfadinformation eine der Pfadinformation 
entsprechendes Symbol umfalit, und das Symbol 



in Verbindung mit der graphischen Tastaturabbil- 
dung angezeigt wird. 

16. Vorrichtung nach Anspruch 14, wobei die Darstel- 
lung der Pfaddarstellung eine Tintenspur umfalit, 
welche der Folge der in der Nahe der graphischen 
Tastaturabbildung angezeigten Punkte, entspricht. 

17. Verfahren nach Anspruch 14, wobei die Darstellung 
der Pfadinformation ein in Verbindung mit der gra- 
phischen Tastaturabbildung angezeigtes Menu um- 
falit. 

18. Verfahren nach Anspruch 1, wobei die Eingangs- 
quelle einen Sensor enthalt, der auf eine Vielzahl 
von Positionen des Zeigegerates in bezug auf den 
Sensor reagiert, wobei der Generator einen mit 
dem Sensor und der Anzeige verbundenen Prozes- 
sor enthalt, um 

20 

in Verbindung mit der Anzeige die graphische 
Tastaturabbildung anzuzeigen, 

von dem Sensor eine die Positionsinformation 
25 und die Pfadinformation umfassende Eingabe- 

grolie zu empfangen, wobei die Abbildung eine 
Vielzahl von Tasten enthalt, jede Taste einer 
Stelle in der Abbildung zugeordnet ist, die Po- 
sitionsinformation eine ausgewahlte Position 
30 von der Vielzahl von Positionen des Zeigege- 

rates umfalit und die Pfadinformation eine Fol- 
ge von Positionen von der Vielzahl von Positio- 
nen des Positionszeigegerates umfalit und die 
Folge die Richtung der Verschiebung darstellt, 

35 

die ausgewahlte Position mit einer Stelle in der 
Abbildung zu korrelieren, um eine ausgewahlte 
Stelle zu bestimmen, 

40 die ausgewahlte Stelle einer Taste von den 

mehreren Tasten zuzuordnen, um eine ausge- 
wahlte Taste zu bestimmen, 

die Pfadinformation zu interpretieren, um einen 
45 Befehl zu bestimmen, und 

ein Ausgabegrdfie zu erzeugen, wobei die Aus- 
gabegrolie aus der Gruppe ausgewahlt wird, 
welche aus einem Zeichen besteht, das durch 
50 Anwenden des Befehls auf die gewahlte Taste 

und ein Zeichen, das durch die Ausfuhrung des 
Befehls unabhangig von der gewahlten Taste 
ermittelt wird, bestimmt wird. 

55 1 9. Vorrichtung nach Anspruch 1 8, wobei ein Puffer mit 
dem Prozessor verbunden ist, um die so erzeugte 
Ausgangsgrolie zu empfangen. 
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20. Vorrichtung nach einem der Anspruche 18 oder 19, 
wobei ein Ereignistreiber mit dem Sensor und dem 
Prozessor verbunden ist, urn als Antwort auf Ver- 
anderungen der relativen Position des Zeigegera- 
tes in bezug auf den Sensor, auf wesentliche Ver- 
anderungen in einer Nahe zwischen dem Zeigege- 
rat und dem Sensor, und auf erhebliche Pausen in 
einer Relativbewegung des Zeigegerates in bezug 
auf den Sensor Ereignisse zu erzeugen, und wobei 
der Prozessor die Ausgangsgrofte als Reaktion auf 
ein von dem Ereignistreiber erzeugtes Ereignis er- 
zeugt. 

21. Vorrichtung nach einem der Anspruche 18 bis 20, 
wobei der Prozessor in Verbindung mit der Anzeige 
eine optische Darsteilung der Pfadinformation in 
Verbindung mit der graphischen Tastaturabbildung 
bereitstellt. 

22. Vorrichtung nach Anspruch 1 , wobei die Anzeige ei- 
nen im wesentlichen ebenen Bildschirm umfaflt, urn 
eine graphische Tastaturabbildung darzustellen, 
welche eine Vielzahl von Tasten umfafit, wobei das 
Zeigegerat ein Schreibgerat umfalit, urn eine Viel- 
zahl von Positionen in bezug auf die graphische Ta- 
staturabbildung zu bezeichnen, die Eingabequelle 
einen mit der Anzeige verbundenen Sensor enthalt, 
urn von dem Schreibgerat bezeichnete Positionen 
zu erfassen, einen Puffer, urn eine Ausgangsgrofce 
zu erfassen und der Generator einen Prozessor 
enthalt, der mit dem Anzeigebildschirm, dem Sen- 
sor und dem Puffer verbunden ist, urn ein Zeichen 
als Reaktion auf eine Folge von dem Schreibgerat 
bezeichneter Positionen zu erzeugen, und die Fol- 
ge von Positionen die Verschiebungsrichtung als ei- 
ne Relativbewegung des Schreibgerates in bezug 
auf die graphische Tastaturabbildung und im we- 
sentlichen in einer Oberflache, welche der Ebene 
des Anzeigebildschirms entspricht, beschreibt. 

23. Vorrichtung nach Anspruch 22, wobei der Anzeige- 
bildschirm und der Sensor physikalisch in einer Ein- 
gabe-Erfassungs-Anzeige verbunden sind, und die 
Oberflache im wesentlichen eben und wesentlichen 
parallel zu der Ebene des Anzeigebildschirms liegt. 

24. Vorrichtung nach einem der Anspruche 22 oder 23, 
wobei der Sensor uber dem Anzeigebildschirm 
liegt. 

25. Vorrichtung nach einem der. Anspruche 22 bis 24, 
wobei der Sensor unter dem Anzeigebildschirm 
liegt. 

26. Verfahren zurAnwendung einer graphischen Tasta- 
tur mit einem Schritt einer Darsteilung einer graphi- 
schen Tastaturabbildung uber eine Anzeige und mit 
den Schritten: 



Verwenden einer ein Zeigegerat umfassenden 
Eingangsquelle, , urn eine Tasteninformation 
und Pfadinformation umfassende Eingangs- 
grolie zu erzeugen, wobei die Tasteninformati- 
on einer Taste entspricht, die auf der graphi- 
schen Tastaturabbildung durch das Zeigegerat 
bezeichnet wird, und die Pfadinformation min- 
destens einer Verschiebungsrichtung des Zei- 
gegerates auf der graphischen Tastaturabbil- 
dung entspricht; und 

Verwenden eines Ausgabegenerators, urn als 
Reaktion auf die Tasteninformation und die 
Pfadinformation ein Ausgangszeichen definie- 
rende Tastencodedaten auszugeben, indem 
ein der Pfadinformation entsprechender Modi- 
fikator auf ein der Tasteninformation entspre- 
chendes Eingabezeichen angewendet wird, 
und der Modifikator gemaft der mindestens ein 
Verschiebungsrichtung bestimmt wird. 

27. Verfahren nach Anspruch 26, wobei der Schritt der 
Verwendung des Ausgabegenerators zum Erzeu- 
gen einer Ausgangsgro&e als Reaktion auf die 
Pfadinformation femerdie Schritte der Verwendung 
des Ausgangsgenerators zur Prufung der Folge von 
Punkten zur Bestimmung eines Satzes von Punk- 
ten beinhaltet, urn einen Satz von Knickpunkten zu 
bestimmen, und der Ausgangsgenerator dazu ver- 
wendet wird, einen Befehl gemali dem Satz der 
Knickpunkte zu bestimmen. 

28. Verfahren nach Anspruch 26, welches ferner die 
Schritte der Bereitstellung eines einen Prozessors, 
einen Sensor, eine Anzeige und einen Puffers be- 
inhaltenden Computersystems, und eines Verfah- 
rens zur Verwendung einer graphischen Tastatur 
zum Erzeugen eines Zeichens umfalit, wobei der 
Schritt der Verwendung einer Eingangsquelle den 
Schritt einer Erfassung einer Bewegung mit dem 
Sensor umfafit, welche der mindestens einen Ver- 
schiebungsrichtung des Zeigegerates in bezug auf 
den Sensor entspricht, indem das Zeigegerat in un- 
mittelbarer Nahe zu dem Sensor uber einen erheb- 
lichen Teil der Bewegung gehalten wird und in dem 
Puffer der so erzeugte Tastencode empfangen wird. 

29. Verfahren nach Anspruch 28, wobei der Schritt der 
Verwendung des Prozessors zum Erzeugen des 

50 Zeichens die Schritte der Bestimmung eines unmo- 
difizierten Zeichens gemafc einer Anfangsposition 
des Zeigegerates an einem Startpunkt der Bewe- 
gung, der Bestimmung eines Modifikators gemaft 
einem Pfad, der von dem Zeigegerat wahrend der 

55 Bewegung genommen wird und der Anwendung 
des Modifikators auf das unmodifizierte Zeichen be- 
inhaltet. 
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30. Verfahren nach Anspruch 28, wobei der Schritt der 
Verwendung des Prozessors zum Erzeugen des 
Zeichens die Ermittlung eines speziellen Zeichens 
anhand eines Pfades, welcher von dem Zeigegerat 
wahrend der Bewegung bezeichnet wird und unab- 
hangig von einer Anfangsposition des Schreibgera- 
tes an einem Startpunkt der Bewegung beinhaltet. 

31. Verfahren nach Anspruch 26, welches femer die 
Schritte der Darstellung eines Menus mit einer Viel- 
zahl von Elementen, der Bereitstellung der graphi- 
schen Tastaturabbildung mit einer Darstellung einer 
Vielzahl von Tasten unter Verwendung des Zeige- 
gerates zum Bezeichnen eines Elementes aus dem 
Menu und der Verwendung des Prozessor zum Er- 
zeugen eines Zeichens gemaft dem bezeichneten 
Menuelement umfa&t, wobei die einer Taste aus der 
Vielfalt von Tasten, welche von dem Zeigegerat be- 
zeichnet wird, entsprechendeTasteninformation ei- 
ne bezeichnete Taste definiert. 

32. Verfahren nach Anspruch 31 , wobei der Schritt der 
Anzeige eines Menus die Anzeige eines radialen 
Menus mit einer Darstellung eines Satzes von Mo- 
difikatoren umfafit, wobei der Schritt der Verwen- 
dung des Zeigegerates zum Bezeichnen eines Ele- 
mentes einen Schritt einer Auswahl eines Modifika- 
tors aus einem Satz von Modifikatoren umfa&t, und 
der Schritt der Verwendung des Prozessor zum Er- 
zeugen des Zeichens einen Schritt einer Anwen- 
dung des ausgewahlten Modifikators auf ein der be- 
zeichneten Taste entsprechendes Zeichen umfalM. 

33. Verfahren nach Anspruch 31 , wobei der Schritt der 
Anzeige eines Menus die Anzeige eines radialen 
Menus mit einer Darstellung eines Satzes speziel- 
ler Zeichen umfafM, der Schritt der Verwendung des 
Zeigegerates zum Bezeichnen eines Elementes ei- 
nen Schritt der Auswahl ein speziellen Zeichens 
aus dem Satz der speziellen Zeichen umfaftt, und 
der Schritt der Verwendung des Prozessor zum Er- 
zeugen des Zeichens einen Schritt der Erzeugung 
des ausgewahlten speziellen Zeichens unabhangig 
von der bezeichneten Taste umfaftt. 
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1 . Appareil d'une interface utilisateur pour un systeme 
d'ordinateurs (1) comportant 

un dispositif d'affichage (20) pour afficher une 
image du clavier graphique (25), 

une source d'entree comprenant un dispositif 
de designation (10), ladite source d'entree de- 
livre une entree comprenant des informations 
de touche correspondant a une touche desi- 



gnee sur ladite image du clavier graphique par 
ledit dispositif de designation et des informa- 
tions de trajet correspondant au moins a une 
direction de deplacement dudit dispositif de de- 
5 signation sur ladite image du clavier graphique, 

et 

un generateur de sortie, repondant a des infor- 
mations de trajet delivrees par ladite source 

10 d'entree, pour produire des donnees de code 

de touche definissant un caractere de sortie en 
appliquant un modificateur correspondant 
auxdites informations de trajet a un caractere 
d'entree correspondant auxdites informations 

15 de touche, avec ledit modificateur etant deter- 

mine en conformite avec ladite au moins une 
direction de deplacement. 

2. Appareil selon la revendication 1 , dans lequel ladite 
20 image du clavier graphique (25) comprend une re- 
presentation d'une pluralite de caracteres, et lesdi- 
tes informations de touche delivrees par ladite sour- 
ce d'entree comprend un caractere auquel ladite 
touche correspond. 

Appareil selon I'une quelconque des revendications 
1 ou 2, dans lequel ladite source d'entree com- 
prend, en outre, un capteur (22) sensible a une plu- 
ralite de positions dudit dispositif de designation 
30 (10), ledit capteur delivrant a ladite source d'entree 
des informations de position concernant ledit dispo- 
sitif de designation (10) pendant une manipulation 
dudit dispositif de designation (10), lesdites infor- 
mations de position determinant ladite direction de 
35 deplacement. 

4. Appareil selon la revendication 3, dans lequel ledit 
capteur (22) est integre audit affichage (20), lesdi- 
tes informations de position incluent des donnees 

40 de position au voisinage de I'image du clavier gra- 
phique sur une partie substantielle de ladite mani- 
pulation dudit dispositif de designation (10). 

5. Appareil selon I'une quelconque des revendications 
45 1 a 4, dans lequel ledit dispositif de designation (10) 

comprend un dispositif de pointage selectionne 
dans le groupe qui est constitue d'un stylo, d'un sty- 
lo optique, d'un stylet, d'un crayon, d'un pointeur, 
d'une souris, d'une boule roulante, d'une manette 
50 de jeu ou d'un doigt. 

6. Appareil selon I'une quelconque des revendications 
1 a 5, dans lequel ledit dispositif de designation 
comprend un curseur affiche par ledit dispositif d'af- 

55 fichage (20), ledit curseur procurant une represen- 
tation d'une position d'une pluralite de positions. 

7. Appareil selon I'une quelconque des revendications 
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1 a 6, dans lequel (edit modificateur est selectionne 
dans le groupe qui est constitue de shift, majuscule, 
control, alt, option et command. 

8. Appareil selon I'une quelconque des revendications 
1 a 7, dans lequel ledit generateur de sortie produit 
ladite sortie en determinant un caractere corres- 
pondant auxdites informations de trajet et indepen- 
damment desdites informations de touche. 

9. Appareil selon la revendication 8, ledit caractere est 
un caractere special choisi dans le groupe qui est 
constitue de return, space, backspace et delete. 

1 0. Appareil selon I'une quelconque des revendications *5 
1 a 9, comprenant, en outre, une memoire tampon 
(30) pour memoriser ledit caractere de sortie. 

1 1 . Appareil selon I'une quelconque des revendications 

1 a 10, ladite source d'entree delivrant desdites in- 20 
formations de trajet en conformite avec une position 
dudit dispositif de designation (1 0) par rapport a la- 
dite image du clavier graphique (25). 

1 2. Appareil selon I'une quelconque des revendications 25 
1 a 11, ladite source d'entree delivrant lesdites in- 
formations de touche en conformite avec un point 
d'une serie de points correspondant au deface- 
ment dudit dispositif de designation. 

30 

13. Appareil selon la revendication 12, ledit point etant 
un point de debut de ladite serie. 

14. Appareil selon I'une quelconque des revendications 

1 a 1 3, ledit dispositif d'affichage affichant, en outre, 35 
une representation desdites informations de trajet. 

15. Appareil selon la revendication 14, dans lequel la- 
dite representation desdites informations de trajet 
comprend une marque correspondant auxdites in- 40 
formations de trajet, ladite marque etant affichee en 
liaison avec ladite image du clavier graphique. 

16. Appareil selon la revendication 14, ladite represen- 
tation desdites informations de trajet comprenant 
une trainee d'encre correspondant a ladite serie 
des points affiches au voisinage de ladite image du 
clavier graphique. 

17. Appareil selon la revendication 14, ladite represen- so 
tation desdites informations de trajet comprenant 

un menu affiche en liaison avec ladite image du cla- 
vier graphique. 

18. Appareil selon la revendication 1 , dans lequel ladite 55 
source d'entree inclut un capteur, sensible a une 
pluralite de positions dudit dispositif de designation 
par rapport audit capteur, avec ledit generateur in- 



cluant un processeur, couple audit capteur et ledit 
affichage, pour 

afficher, en liaison avec ladite affichage, ladite 
image du clavier graphique, 

recevoir depuis ledit capteur une entree com- 
prenant aux informations de position et lesdites 
informations de trajet, avec ladite image in- 
cluant une pluralite de touches, chacune de ces 
touches etant associee a un emplacement 
dans ladite image, lesdites informations de po- 
sition comprenant une position selectionnee de 
ladite pluralite de positions dudit dispositif de 
designation, et lesdites informations de trajet 
comprenant une serie de positions de ladite 
plurality des positions dudit designateur de po- 
sition, ladite serie representant ladite direction 
de deplacement, 

correler ladite position selectionnee avec un 
emplacement dans ladite image pour determi- 
ner un emplacement selectionne, 

associer ledit emplacement selectionne a une 
touche de ladite pluralite de touches pour de- 
terminer une touche selectionnee, 

interpreter lesdites informations de trajet pour 
determiner une commande et 

generer une sortie, ladite sortie etant selection- 
nee a partir du groupe qui est constitue d'un ca- 
ractere determine par I'application de ladite 
commande a ladite touche selectionnee et d'un 
caractere determine par I'intermediaire de 
l'execution de ladite commande independam- 
ment de ladite touche selectionnee. 

19. Appareil selon la revendication 18, dans lequel une 
memoire tampon est couplee audit processeur pour 
recevoir ladite sortie ainsi generee. 

20. Appareil selon I'une quelconque des revendications 
18 ou 19, dans lequel un circuit de commande 
d'evenements est couple audit capteur et audit pro- 
cesseur, pour generer des evenements a des chan- 
gements sensibles dans ladite position relative du- 
dit dispositif de designation par rapport audit cap- 
teur, a des changements sensibles a proximite en- 
tre ledit dispositif de designation et ledit capteur et 
a des arrets substantiels dans un mouvement relatif 
dudit dispositif de designation par rapport audit cap- 
teur et dans lequel ledit processeur genere ladite 
sortie en reponse a un evenement genere par ledit 
circuit de commande d'evenements. 

21 . Appareil selon I'une quelconque des revendications 
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18 a 20, dans lequel ledit processeur, en liaison 
avec ledit affichage, delivre une representation vi- 
suelle desdites informations de trajet en liaison 
avec ladite image du clavier graphique. 

22. Appareil selon la revendication 1 , dans lequel ledit 
affichage comprend un ecran sensiblement plan 
pour afficher une image du clavier graphique com- 
prenant une pluralite de touches, avec ledit dispo- 
sitif de designation incluant un stylo pour designer 
une pluralite de positions par rapport a ladite image 
du clavier graphique, ladite source d'entree incluant 
un capteur, couple audit affichage, pour detecter les 
positions designees par ledit stylo, une memoire 
tampon pour recevoir une sortie et ledit generateur 
incluant un processeur, couple audit ecran d'affi- 
chage, audit capteur et a ladite memoire tampon, 
pour generer un caractere en reponse a une serie 
de positions designee par ledit stylo, ladite serie de 
positions decrivant ladite direction de deplacement 
comme mouvement relatif dudit stylo par rapport a 
ladite image du clavier graphique et sensiblement 
dans une surface correspondant au plan dudit 
ecran d'affichage. 

23. Appareil selon la revendication 22, dans lequel ledit 
ecran d'affichage et ledit capteur sont physique- 
ment couples dans un affichage de detection d'en- 
tree, avec ladite surface etant sensiblement plane 
et sensiblement paraliele audit plan dudit ecran 
d'affichage. 

24. Appareil selon I'une quelconque des revendications 
22 ou 23, dans lequel ledit capteur recouvre ledit 
ecran d'affichage. 

25. Appareil selon I'une quelconque des revendications 
22 a 24, dans lequel ledit capteur se trouve au-des- 
sous de I'ecran d'affichage. 

26. Procede pour utiliser un clavier graphique compor- 
tant une etape consistant a afficher une image du 
clavier graphique via un affichage et comprenant 
les etapes consistant a : 

utiliser une source d'entree comprenant un dis- 
positif de designation pour delivrer une entree 
comprenant des informations de touche et des 
informations de trajet, lesdites informations de 
touche correspondant a une touche designee 
sur ladite image du clavier graphique par ledit 
dispositif de designation, lesdites informations 
de trajet correspondant a au moins une direc- 
tion de deplacement dudit dispositif de desi- 
gnation sur ladite image du clavier graphique ; 
et 

utiliser un generateur de sortie pour produire, 



en reponse auxdites informations de touche et 
auxdites informations de trajet, des donnees de 
code de touche definissant un caractere de sor- 
tie en appliquant un modificateur correspon- 
5 dant auxdites informations de trajet a un carac- 

tere d'entree correspondant auxdites informa- 
tions de touche, avec ledit modificateur etant 
determine en conformite avec au moins une di- 
rection de deplacement. 

10 

27. Procede selon la revendication 26, dans lequel la- 
dite etape consistant a utiliser ledit generateur de 
sortie pour produire une sortie en reponse auxdites 
informations de trajet inclut, en outre, les etapes 

15 consistant a utiliser ledit generateur de sortie pour 
examiner ladite serie de points pour determiner un 
ensemble de points d'articulation et utiliser ledit ge- 
nerateur de sortie pour determiner une commande 
en conformite avec ledit ensemble de points d'arti- 

20 culation. 

28. Procede selon la revendication 26, comprenant, en 
outre, les etapes consistant a procurer un systeme 
d'ordinateurs incluant un processeur, un capteur, un 

25 affichage et une memoire tampon, un procede pour 
utiliser un clavier graphique pour produire un carac- 
tere, dans lequel ladite etape consistant a utiliser 
une source d'entree comprend une etape consis- 
tant a detecter avec ledit capteur un mouvement 

30 correspondant a au moins une direction de depla- 
cement dudit dispositif de designation par rapport 
audit capteur en maintenant ledit dispositif de desi- 
gnation a proximite dudit capteur sur une partie 
substantielle dudit mouvement, et recevoir dans la- 

35 dite memoire tampon ledit code de touche ainsi ge- 
nere. 

29. Procede selon la revendication 28, dans lequel la- 
dite etape consistant a utiliser ledit processeur pour 

40 generer ledit caractere inclut les etapes consistant 
a determiner un caractere non modifie en conformi- 
te avec une position initiale dudit dispositif de desi- 
gnation a un point du debut dudit mouvement, de- 
terminer un modificateur en conformite avec un tra- 

45 jet pris par ledit dispositif de designation pendant 
ledit mouvement et appliquer ledit modificateur 
audit caractere non modifie. 

30. Procede selon la revendication 28, dans lequel la- 
50 dite etape consistant a utiliser processeur pour ge- 
nerer ledit caractere inclut I'etape consistant a de- 
terminer un caractere special en conformite avec 
un trajet suivi par ledit dispositif de designation pen- 
dant ledit mouvement et independamment d'une 

55 position initiale dudit dispositif de stylo a un point 
du debut dudit mouvement. 

31. Procede selon la revendication 26, comprenant, en 
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outre, les etapes consistant a afficher un menu 
comportant une pluralite d'elements, a fournir ladite 
image du clavier graphique avec une representa- 
tion d'une pluralite de touches, a utiliser ledit dispo- 
sitif de designation pour designer un element a par- 5 
tir dudit menu et a utiliser ledit processeur pour ge- 
nerer un caractere en conformite avec ledit element 
de menu designe, avec lesdites informations de 
touche correspondant a une touche parmi ladite 
pluralite des touches designees par ledit dispositif *o 
de designation, definissant une touche designee. 

32. Procede selon la revendication 31, dans lequel. la- 
dite etape consistant a afficher un menu comprend 
I'affichage d'un menu radial comprenant une repre- ™ 
sentation d'un ensemble de modificateurs, ladite 
etape d'utilisation dudit designates pour designer 

un element comprend une etape consistant a se- 
lectionner un modificateur a partir dudit ensemble 
de modificateurs et ladite etape consistant a utiliser 20 
ledit processeur pour generer ledit caractere inclut 
une etape consistant a appliquer ledit modificateur 
selectionne a un caractere correspondant a ladite 
touche designee. 

25 

33. Procede selon la revendication 31, dans lequel la- 
dite etape consistant a afficher un menu comprend 
I'affichage d'un menu radial comprenant une repre- 
sentation d'un ensemble de caracteres speciaux, 
ladite etape d'utilisation dudit designates pour de- 30 
signer un element comprend une etape consistant 

a selectionner un caractere special a partir dudit en- 
semble de caracteres speciaux, et ladite etape d'uti- 
lisation dudit processeur pour generer ledit carac- 
tere comprend une etape consistant a produire ledit 35 
caractere special selectionne independamment de 
ladite touche designee. 
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